JP7605263B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

Currently, there is known technology that simulates the state of a user's face when cosmetics are applied to the face. For example, Patent Document 1 describes a technology that superimposes makeup parts representing cosmetics on a user's facial image, and changes the makeup parts superimposed on the facial image depending on the state of the eyes. Specifically, Patent Document 1 describes a technology that transforms makeup parts drawn on an image with closed eyes for an image with open eyes, and superimposes the transformed makeup parts on the image with open eyes.

Here, in actual makeup, the way in which the appearance of cosmetics applied to the face changes with changes in the state of the eyes differs depending on the type of upper eyelid. For example, even if a person with single eyelids and a person with double eyelids apply eyeshadow to the same area with their eyes closed, the area of applied eyeshadow that is visible when the person opens their eyes will be different for the person with single eyelids and the person with double eyelids. In this regard, Patent Document 1 describes taking the type of upper eyelid into consideration when changing the makeup parts superimposed on the face image according to the state of the eyes.

International Publication No. 2018/159231

However, in the method described in Patent Document 1, the way in which the makeup parts are changed does not match the type of upper eyelid, and there is a possibility that the state of the face with cosmetics applied will not be properly simulated. For example, Patent Document 1 does not distinguish between single eyelids and deep-set double eyelids, but it is considered preferable to distinguish between single eyelids and deep-set double eyelids. Patent Document 1 also describes that when the upper eyelid is single eyelid, the upper eyelid makeup parts drawn on the image with the eyes closed are superimposed on the image with the eyes open without being deformed, but it is considered preferable to deform the upper eyelid makeup parts. For this reason, a technology is desired that can properly simulate the state of the face with cosmetics applied according to the type of facial parts.

The present invention has been made in consideration of the above problems, and aims to provide an image processing device, an image processing method, and a program that appropriately simulate the state of a face to which cosmetics have been applied according to the type of facial feature.

In order to achieve the above object, the image processing device according to the present invention comprises:
an image acquisition means for acquiring an open-eye image, which is a facial image of a user captured when the upper eyelid is open ;
The device further includes an area determining means for determining, when determining a makeup area on the eye-open image of the user based on the makeup action performed by the user when the upper eyelid is closed and the type of the upper eyelid , the makeup area when the user has a single eyelid is larger than the makeup area when the user has a deep double eyelid .

According to the present invention, it is possible to appropriately simulate the state of a face after applying cosmetics according to the type of facial feature.

FIG. 1 is an external view of a makeup simulation device according to a first embodiment of the present invention; FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus according to a first embodiment of the present invention. FIG. 1 is a block diagram showing a hardware configuration of an instruction device according to a first embodiment of the present invention. FIG. 1 is a block diagram showing a functional configuration of a makeup simulation device according to a first embodiment of the present invention. FIG. 1 is a diagram showing a state in which a user is using an image processing apparatus according to a first embodiment of the present invention; FIG. 1 is a diagram showing a state in which a user uses a pointing device to operate a touch screen included in an image processing device according to a first embodiment of the present invention. A diagram showing various lines detected near the eye. 1A is a diagram showing characteristics of each type of upper eyelid, (A) is a diagram showing characteristics of a single eyelid, (B) is a diagram showing characteristics of a double eyelid, and (C) is a diagram showing characteristics of a deep double eyelid. A diagram showing a user applying makeup while closing their eyes 1A is a diagram showing an eye-opening makeup area for a single eyelid, FIG. 1B is a diagram showing an eye-opening makeup area for a double eyelid, and FIG. 1C is a diagram showing an eye-opening makeup area for a deep double eyelid. 10 is a flowchart showing an image display process executed by the image processing device according to the first embodiment of the present invention. A flowchart showing the type determination process in FIG. Flowchart showing the region determination process in FIG. 11 FIG. 11 is a block diagram showing the functional configuration of an image processing apparatus according to a second embodiment of the present invention.

Embodiments of the present invention will be described below with reference to the drawings. Note that in the drawings, the same or corresponding parts are given the same reference numerals.

(Embodiment 1)
First, the configuration of a makeup simulation device 100 according to a first embodiment of the present invention will be described with reference to Fig. 1. The makeup simulation device 100 is a device that simulates on a computer what the user's appearance will be after applying makeup with cosmetics. By using the makeup simulation device 100, the user can easily practice applying makeup and check in advance what the user's appearance will be after applying makeup, without using real cosmetics.

As shown in FIG. 1, the makeup simulation device 100 includes an image processing device 10 and an instruction device 30. The image processing device 10 is a device that uses a captured image of a user's face to generate and display an image representing the state after makeup is applied by simulation. The image processing device 10 has a shape that imitates, for example, a makeup palette.

Figure 2 shows the hardware configuration of the image processing device 10. As shown in Figure 2, the image processing device 10 includes a processor 11, a flash memory 12, a camera 13, a first touch screen 14, a second touch screen 15, and a communication interface 16. These elements included in the image processing device 10 are connected to each other via, for example, a system bus.

The processor 11 controls the overall operation of the image processing device 10. The processor 11 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and an RTC (Real Time Clock). The flash memory 12 stores various types of information.

The camera 13 captures an image of a subject. The camera 13 includes a lens that collects light emitted from the subject, an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) that receives the collected light and captures an image of the subject, and an A/D (Analog/Digital) converter that converts the imaging data sent from the imaging element as an electrical signal into digital data. As shown in FIG. 1, the camera 13 is disposed near the first touch screen 14, and captures an image of the face of a user who is using the image processing device 10 while looking at the first touch screen 14.

The first touch screen 14 and the second touch screen 15 are user interfaces that accept various information from the user and present various information to the user. As shown in FIG. 1, the first touch screen 14 is disposed on the upper surface of the image processing device 10 and displays an image captured by the camera 13. The second touch screen 15 is disposed on the lower surface of the image processing device 10 and displays a cosmetic image that represents a cosmetic to be virtually applied to the user's face in the makeup simulation. The second touch screen 15 also accepts user operations that involve contact with the surface of the second touch screen 15.

The communication interface 16 is a module that communicates with external devices including the instruction device 30 under the control of the processor 11. The communication interface 16 communicates with external devices, for example, via wireless communication such as a wireless LAN (Local Area Network) or Bluetooth (registered trademark), or via wired communication such as a USB (Universal Serial Bus) cable.

The pointing device 30 is a device for selecting cosmetics in a makeup simulation and indicating the position on the face where the selected cosmetics will be virtually applied. The pointing device 30 has a stick-like shape and is a component that imitates an application tool used to apply cosmetics to the face, such as a makeup brush. The user holds the pointing device 30 in their hand and uses it by pointing the end of the pointing device 30 at the cosmetic image displayed on the image processing device 10 and the user's actual face.

Figure 3 shows the hardware configuration of the indication device 30. The indication device 30 includes a processor 31, a flash memory 32, a pressure sensor 33, and a communication interface 34. These elements of the indication device 30 are connected to each other via, for example, a system bus.

The processor 31 controls the overall operation of the indication device 30. The processor 31 includes, for example, a CPU, a ROM, a RAM, and an RTC. The flash memory 32 stores various information. The pressure sensor 33 detects pressure applied to the end of the indication device 30. In other words, the pressure sensor 33 detects contact with the end of the indication device 30. The pressure sensor 33 detects pressure using a method such as a capacitance method or a strain gauge method. The communication interface 34 is a module that communicates with the image processing device 10 under the control of the processor 31. The communication interface 34 communicates with the image processing device 10 by wireless communication such as a wireless LAN or Bluetooth (registered trademark), or wired communication such as a USB cable.

Next, the functions of the makeup simulation device 100 will be described with reference to FIG. 4. The image processing device 10 functionally includes an imaging unit 101, an image acquisition unit 102, a display control unit 103, a display unit 104, an operation reception unit 105, a cosmetics selection unit 106, an information receiving unit 107, a contact position identification unit 108, a state determination unit 109, a type determination unit 110, an area determination unit 113, a makeup part generation unit 114, and a makeup part holding unit 115. The instruction device 30 functionally includes a contact detection unit 301 and an information transmission unit 302.

The image acquisition unit 102 is an example of an image acquisition means in the present invention. The display control unit 103 is an example of a display control means in the present invention. The display unit 104 is an example of a display means in the present invention. The operation reception unit 105 is an example of an operation reception means in the present invention. The contact position identification unit 108 is an example of a contact position identification means in the present invention. The type discrimination unit 110 is an example of a type discrimination means in the present invention. The area determination unit 113 is an example of an area determination means in the present invention.

The imaging unit 101 captures an image of an area including the user's face and generates an image representing this area. FIG. 5 shows a state in which a user faces a first touch screen 14 provided in the image processing device 10 to simulate makeup using the makeup simulation device 100. The imaging unit 101 captures an image of the area inside the field of view indicated by the dashed line in FIG. 5, specifically, an area including the face of the user facing the first touch screen 14.

The installation position and the direction of the optical axis of the camera 13 for realizing the imaging unit 101 are therefore adjusted so that the face of the user facing the first touch screen 14 is included in the field of view of the camera 13. The imaging unit 101 continuously executes a process of capturing images of the user's face and generating a captured image while the image processing device 10 is being used by the user. The function of the imaging unit 101 is realized, for example, by the processor 11 and the camera 13 working together.

The image acquisition unit 102 acquires the captured image generated by the imaging unit 101. This captured image is either a first face image or a second face image. The first face image is a face image of the user captured when the facial parts are in a first state. The second face image is a face image of the user captured when the facial parts are in a second state. This facial part is, for example, one of the parts included in the face whose state changes significantly.

In this embodiment, the eye to which makeup is applied is the left eye, and the facial feature is the upper eyelid of the left eye. Hereinafter, the upper eyelid of the left eye is simply referred to as the upper eyelid, where appropriate. The second state is a state different from the first state. For example, when the first state is a closed state, the second state is an open state, and when the first state is an open state, the second state is a closed state. In this embodiment, the first state is a closed state, and the second state is an open state. In this embodiment, the first face image is a captured image of the user's face captured when the user has his/her left eye closed, and the second face image is a captured image of the user's face captured when the user has his/her left eye open. Hereinafter, the first face image is referred to as an image of closed eyes, and the second face image is referred to as an image of open eyes, where appropriate. The function of the image acquisition unit 102 is realized, for example, by the function of the processor 11.

The display control unit 103 controls the display of the image by the display unit 104. Specifically, the display control unit 103 displays the captured image acquired by the image acquisition unit 102 in a first display area provided in the display unit 104. The first display area is, for example, a display area corresponding to the first touch screen 14. In this way, the display control unit 103 displays the appearance of the user's face in real time. Therefore, the user can simulate makeup application while checking the image of his or her own face displayed in the first display area provided in the display unit 104 as if it were his or her face reflected in a mirror.

The display control unit 103 also displays a cosmetic image representing the cosmetic in a second display area provided on the display unit 104. The second display area is a display area different from the first display area, and is, for example, a display area corresponding to the second touch screen 15. The cosmetic image is an image that visually represents the cosmetic virtually applied to the user's face in the makeup simulation. As shown in FIG. 6, the display control unit 103 displays images representing multiple cosmetic products as the cosmetic image. Note that FIG. 6 shows how the user operates the second touch screen 15 provided on the image processing device 10 using the pointing device 30.

Here, each of the multiple cosmetics differs in at least one aspect, such as color, gloss, or texture. Images representing the multiple cosmetics are displayed in different areas within the second display area. The user can refer to the cosmetic image and select from the multiple cosmetics the cosmetic for which they wish to perform a makeup simulation and use it. The images representing the cosmetic images are stored in the flash memory 12. In this way, a captured image representing the user's face is displayed in the first display area, and the cosmetic image is displayed in the second display area, allowing the user to treat the image processing device 10 as a palette of cosmetics. In this embodiment, the cosmetics are eye shadows. The functions of the display control unit 103 are realized, for example, by the functions of the processor 11.

The display unit 104 displays an image according to the control of the display control unit 103. Specifically, the display unit 104 displays a first face image or a second face image in the first display area, and displays a cosmetic image in the second display area. The function of the display unit 104 is realized, for example, by the processor 11, the first touch screen 14, and the second touch screen 15 working together.

The operation acceptance unit 105 accepts a user operation involving contact with the second display area using the pointing device 30 when the display unit 104 is displaying a cosmetic image in the second display area. FIG. 6 shows a state in which a user touches the tip of the pointing device 30 held in the hand to the surface of the second touch screen 15. The action of touching the tip of the pointing device 30 to the surface of the second touch screen 15 is an action simulating the action of applying cosmetics stored in a cosmetic palette to the tip of a cosmetic brush. The operation acceptance unit 105 transmits, for example, contact position information indicating the contact position on the second display area to the cosmetic selection unit 106. The function of the operation acceptance unit 105 is realized, for example, by cooperation between the processor 11 and the second touch screen 15.

The cosmetics selection unit 106 selects cosmetics to be used in the makeup simulation based on the user operation received by the operation reception unit 105. Specifically, the cosmetics selection unit 106 identifies the cosmetics selected by the user based on the contact position information received from the operation reception unit 105 and the arrangement information stored in the flash memory 12. The arrangement information is information that indicates the arrangement of each cosmetic product in the second display area, and is information that indicates the coordinates of the area where an image representing the cosmetic product is displayed. The function of the cosmetics selection unit 106 is realized, for example, by the function of the processor 11.

The contact detection unit 301 detects whether or not an object is in contact with the tip of the pointing device 30. The contact detection unit 301 detects pressure applied to the tip of the pointing device 30, and determines whether or not there is contact based on the detected pressure. For example, when the contact detection unit 301 detects contact, it transmits contact information indicating the contact time to the information transmission unit 302. The function of the contact detection unit 301 is realized, for example, by the processor 31 and the pressure sensor 33 working together.

The information transmission unit 302 transmits the contact information generated by the contact detection unit 301 to the image processing device 10. The function of the information transmission unit 302 is realized, for example, by the processor 31 and the communication interface 34 working together.

The information receiving unit 107 receives contact information from the instruction device 30. The function of the information receiving unit 107 is realized, for example, by the processor 11 and the communication interface 16 working together.

When the instruction device 30 comes into contact with the user's face while the display unit 104 is displaying the first face image after the operation receiving unit 105 receives a user operation, the contact position identifying unit 108 identifies the contact position on the user's face where the instruction device 30 has come into contact based on the first face image. When the tip of the instruction device 30 is in contact with an object and the instruction device 30 overlaps with the user's face on the first face image, the contact position identifying unit 108 determines that the instruction device 30 has come into contact with the user's face.

The contact position identification unit 108 can identify the period during which the tip of the pointing device 30 is in contact with some object based on the contact information supplied from the information receiving unit 107. The contact position identification unit 108 can also identify the contact position from the position of the pointing device 30 in the first face image acquired during the period during which the pointing device 30 is in contact with the user's face. The function of the contact position identification unit 108 is realized, for example, by the function of the processor 11.

The state determination unit 109 determines the state of a specific facial part of the user's face. Specifically, the state determination unit 109 determines whether the state of the upper eyelid of the user's left eye is closed or open, in other words, whether the captured image acquired by the image acquisition unit 102 is a first face image or a second face image. The state of the upper eyelid can also be said to be the state of the eyes. Therefore, the state determination unit 109 determines whether the user's left eye in the captured image is closed or open. The method of determining the state of the eyes can be adjusted as appropriate.

For example, the state determination unit 109 identifies the position of the face image in the captured image, identifies the position of the left eye image, which is an image near the left eye, in the face image, and determines whether or not a circular image representing an pupil is present in the left eye image. If the state determination unit 109 determines that there is no circular image, it determines that the state of the upper eyelid of the user's left eye is closed. If the state determination unit 109 determines that there is a circular image, it determines that the state of the upper eyelid of the user's left eye is open. The function of the state determination unit 109 is realized, for example, by the function of the processor 11.

The type discrimination unit 110 discriminates the type of the user's facial features based on at least one of the first face image and the second face image acquired by the image acquisition unit 102. Specifically, the type discrimination unit 110 discriminates whether the type of the upper eyelid of the user's left eye is single, double, or deep-set eyelid based on at least one of the closed eye image and the open eye image. Note that the type of the upper eyelid being single is synonymous with the upper eyelid being single-set eyelid, the type of the upper eyelid being double is synonymous with the upper eyelid being double-set eyelid, and the type of the upper eyelid being deep-set eyelid is synonymous with the upper eyelid being deep-set eyelid. In this embodiment, the type of the upper eyelid is discriminated from the open eye image. The function of the type discrimination unit 110 is realized, for example, by the function of the processor 11. The type discrimination unit 110 includes a line detection unit 111 and a line discrimination unit 112. The line detection unit 111 is an example of a line detection means in the present invention. The line discrimination unit 112 is an example of a line discrimination means in the present invention.

The line detection unit 111 detects a first line representing the groove of a double eyelid or deep double eyelid from the second face image acquired by the image acquisition unit 102. The line detection unit 111 also detects a second line representing the mucous membrane portion at the boundary between the upper eyelid and the eye from the second face image acquired by the image acquisition unit 102. The method of detecting the first and second lines can be adjusted as appropriate. Below, with reference to FIG. 7, a method of detecting 12 lines from the image around the left eye in the eye-open image and identifying the first and second lines from the 12 lines will be described.

As shown in FIG. 7, a total of 12 lines, lines L1-L12, can be detected from the image around the left eye in the open-eye image. Note that FIG. 7 shows the image around the left eye in the open-eye image displayed on the first touch screen 14. Line L1 is a line that represents the head of the eyebrow. Line L2 is a line that represents the upper boundary of the eyebrow among the lines that extend from the head of the eyebrow to the brow arch. Line L3 is a line that represents the upper boundary of the eyebrow among the lines that extend from the brow arch to the end of the eyebrow. Line L4 is a line that represents the lower boundary of the eyebrow among the lines that extend from the head of the eyebrow to the brow arch. Line L5 is a line that represents the lower boundary of the eyebrow among the lines that extend from the brow arch to the end of the eyebrow.

Line L6 is the line on the mucous membrane side of the frame lines that represent the upper boundary of the eye. Line L6 is the second line. Line L7 is the line on the eyelash side of the frame lines that represent the upper boundary of the eye. Line L8 is the line on the mucous membrane side of the frame lines that represent the lower boundary of the eye. Line L9 is the line on the eyelash side of the frame lines that represent the lower boundary of the eye. Line L10 is a line that represents the groove of a double eyelid or deep double eyelid. Line L10 is the first line. Line L11 is the upper line of the line called the eyehole that extends from the inner corner of the eye to the outer corner of the eye. Line L12 is the lower line of the line called the eyehole that extends from the inner corner of the eye to the outer corner of the eye.

It is preferable to determine these 12 lines in order of ease of detection, taking into consideration, for example, their positional relationship with the other lines. For example, line L2, which indicates the upper boundary of the eyebrows, and line L4, which indicates the lower boundary of the eyebrows, are considered to be relatively easy to detect. Furthermore, lines L6 and L7, which are the two lines indicating the upper boundary of the eye, and lines L8 and L9, which are the two lines indicating the lower boundary of the eye, are considered to be relatively easy to detect. Therefore, it is preferable to detect these six lines first.

Here, the two lines that may be detected in the area between lines L4 and L7 are lines L10 and L11. Therefore, the upper line among the lines present in this area can be detected as line L11, and the lower line among the lines present in this area can be detected as line L10. Note that line L10 is a line that represents a double eyelid or a deep double eyelid. Therefore, line L10 will not be detected in an image capturing the face of a user with single eyelids. The function of the line detection unit 111 is realized, for example, by the function of the processor 11.

If the first line is not detected by the line detection unit 111, the line discrimination unit 112 determines that the user's upper eyelid is single eyelid. Also, if the first line is detected by the line detection unit 111, the line discrimination unit 112 determines that the user's upper eyelid is not single eyelid. Also, based on the first line and the second line detected by the line detection unit 111, the line discrimination unit 112 determines whether the user's upper eyelid is single eyelid, double eyelid, or deep double eyelid.

In other words, when the first line is detected, the line discrimination unit 112 discriminates whether the user's upper eyelid is a double eyelid or a deep double eyelid based on the first line and the second line. Specifically, the line discrimination unit 112 determines that the user's upper eyelid is a double eyelid if the length in the width direction of the face of the gap area between the first line and the second line is equal to or greater than a threshold value. Then, the line discrimination unit 112 determines that the user's upper eyelid is a deep double eyelid if this length is less than the threshold value.

Below, we will explain a method for determining the type of upper eyelid based on the gap area with reference to Figure 8. Figure 8 (A) is a diagram showing the characteristics of a single eyelid. Figure 8 (B) is a diagram showing the characteristics of a double eyelid. Figure 8 (C) is a diagram showing the characteristics of a deep double eyelid.

As shown in FIG. 8(A), if the upper eyelid of the user's left eye is single eyelid, the first line, line L6A, is detected, but the second line, line L10, is not detected. As shown in FIG. 8(B), if the upper eyelid of the user's left eye is double eyelid, the first line, line L6B, and the second line, line L10B, are detected. Also, as shown in FIG. 8(C), if the upper eyelid of the user's left eye is deep double eyelid, the first line, line L6C, and the second line, line L10C, are detected. Note that lines L6A, L6B, and L6C are collectively referred to as lines L6, and lines L10B and L10C are collectively referred to as lines L10. Here, the positional relationship between lines L6 and L10 differs between double eyelids and deep double eyelids, and the shape and size of the gap area between lines L6 and L10 differ. Therefore, the line discrimination unit 112 discriminates whether the upper eyelid is a double eyelid or deep double eyelid based on the shape and size of this gap area.

In this embodiment, for ease of understanding, the area where line L6 and line L10 almost overlap, in other words the area where the distance between line L6 and line L10 is extremely short, is not considered to be a gap area. In other words, strictly speaking, in this embodiment, the gap area is the area between line L6 and line L10 where the distance between line L6 and line L10 is equal to or greater than a threshold value.

As shown in FIG. 8(B), when the upper eyelid is a double eyelid, region E1, which is the gap region, is a region that has a crescent shape that is elongated in the width direction of the face. The length of region E1 in the width direction of the face (hereinafter, simply referred to as "length") is W1. Region E1 is a region enclosed when both ends of line L6B and line L10B are connected. Specifically, region E1 is a region enclosed by four lines: a line connecting one end of line L6B and one end of line L10B, line L6B, a line connecting the other end of line L6B and the other end of line L10B, and line L10B.

On the other hand, as shown in FIG. 8C, when the upper eyelid is a deep double eyelid, the gap area E2 is a triangular area with one vertex extending in the width direction of the face. The length of area E2 in the width direction of the face is W2. Area E2 is an area enclosed when both ends of line L6C and line L10C are connected. Specifically, area E2 is an area enclosed by four lines: a line connecting one end of line L6C and one end of line L10C, line L6C, a line connecting the other end of line L6C and the other end of line L10C, and line L10C. The line discrimination unit 112 preferably determines whether the upper eyelid is a double eyelid or a deep double eyelid by determining whether the detected gap area is similar to area E1 or area E2.

In this embodiment, the fact that the lengths of areas E1 and E2 in the width direction of the face are significantly different is used to determine the upper eyelid. In other words, the line determination unit 112 determines that the upper eyelid is a double eyelid if the length of the gap area is equal to or greater than a predetermined threshold, and determines that the upper eyelid is a deep double eyelid if the length of the gap area is less than this threshold. This threshold is, for example, about half the length of the eye. The function of the line determination unit 112 is realized, for example, by the function of the processor 11.

The area determination unit 113 determines a first makeup area on the first face image based on the makeup action performed by the user while the display unit 104 is displaying the first face image. The first makeup area is an area on which the first makeup parts are superimposed. The first makeup parts are images representing cosmetics applied to the face in a simulated manner, and are images superimposed on the first face image. The makeup action is an action in which the user applies cosmetics to the face in a simulated manner using the instruction device 30. The area determination unit 113 determines the first makeup area on the first face image based on the contact position identified by the contact position identification unit 108. For example, the area determination unit 113 determines an area corresponding to the contact position identified due to the makeup action of the user with the eyes closed as the closed eyes makeup area on the closed eyes image on which the closed eyes makeup parts are superimposed.

The area determination unit 113 also determines a second makeup area on the second face image based on the makeup action and the type of face part determined by the type determination unit 110. The second makeup area is an area different from the first makeup area, and is an area onto which the second makeup part is superimposed. The second makeup part is an image that represents cosmetics applied to the face in a simulated manner, and is an image that is superimposed on the second face image. For example, the area determination unit 113 determines an area obtained by changing the closed-eye makeup area according to the type of face part as the open-eye makeup area.

Specifically, when the type discrimination unit 110 determines that the user's upper eyelid is a single eyelid, the area determination unit 113 determines an area having a first size as the second makeup area. When the type discrimination unit 110 determines that the user's upper eyelid is not a single eyelid, the area determination unit 113 determines an area having a size smaller than the first size as the second makeup area.

More specifically, when the type discrimination unit 110 determines that the user's upper eyelid is a double eyelid, the area determination unit 113 determines an area having a second size smaller than the first size as the second makeup area. When the type discrimination unit 110 determines that the user's upper eyelid is a deep double eyelid, the area determination unit 113 determines an area having a third size smaller than the second size as the second makeup area. In other words, even if the size of the makeup area for closed eyes is the same, the size of the makeup area for open eyes decreases in the order of single eyelid, double eyelid, and deep double eyelid.

Figure 9 shows how a user applies makeup simulated with their eyes closed. Figure 9 also shows how area E3, which is the makeup area for closed eyes, is determined from the contact position detected when the closed eyes image is acquired. Here, area determination unit 113 can determine the makeup area for open eyes based on the makeup area for closed eyes. For example, area determination unit 113 can determine the remaining area after excluding the area corresponding to the eyes from the makeup area for closed eyes as the makeup area for open eyes.

Here, the region determination unit 113 adjusts the size of the eye-opening makeup region depending on the type of upper eyelid. Specifically, the region determination unit 113 adjusts the size of the eye-opening makeup region so that the size of the eye-opening makeup region is largest when the upper eyelid is a single eyelid, is medium-sized when the upper eyelid is a double eyelid, and is smallest when the upper eyelid is a deep double eyelid. For example, the region determination unit 113 reduces the size of the eye-opening makeup region based on the size of the eye-opening makeup region when the upper eyelid is a single eyelid.

Specifically, the region determination unit 113 determines the remaining region, excluding the region corresponding to the eyes, from region E3, which is the makeup region for closed eyes shown in FIG. 9, as region E4, which is the makeup region for opening eyes for single eyelids shown in FIG. 10(A). Furthermore, the region determination unit 113 determines the region obtained by vertically shrinking region E4, which is the makeup region for opening eyes for single eyelids shown in FIG. 10(A), as region E5, which is the makeup region for opening eyes for double eyelids shown in FIG. 10(B). Then, the region determination unit 113 determines the region obtained by vertically shrinking region E5, which is the makeup region for opening eyes for double eyelids shown in FIG. 10(B), as region E6, which is the makeup region for opening eyes for deep-set double eyelids shown in FIG. 10(C).

In this way, the region determination unit 113 determines each region such that region E4 is larger than region E5, and region E5 is larger than region E6. More specifically, the region determination unit 113 determines each region such that the vertical width of region E4 is larger than the vertical width of region E5, and the vertical width of region E5 is larger than the vertical width of region E6. This is because the overlap of the upper eyelids is shallowest for single eyelids, medium depth for double eyelids, and deepest for deep double eyelids. In other words, the area of the upper eyelid that is hidden when the eyes are open is narrowest for single eyelids, medium width for double eyelids, and widest for deep double eyelids.

The region determination unit 113 can also determine the makeup region for closed eyes based on the makeup region for open eyes, taking into account the type of upper eyelid. Specifically, when the makeup regions for open eyes are the same size, the region determination unit 113 determines the makeup region for closed eyes such that the makeup region for closed eyes for a single eyelid is the smallest, the makeup region for closed eyes for a double eyelid is of medium size, and the makeup region for closed eyes for a deep double eyelid is the largest. Specifically, the region determination unit 113 can determine the makeup region for closed eyes by using the reverse procedure to that for determining the makeup region for open eyes.

For example, the region determination unit 113 determines the region obtained by adding a region corresponding to the eyes to region E4, which is a makeup region for closed eyes for single eyelids, as region E3, which is a makeup region for open eyes. The region determination unit 113 also determines the region obtained by slightly extending region E5, which is a makeup region for closed eyes for double eyelids, in the vertical direction and adding a region corresponding to the eyes to that region, as region E3, which is a makeup region for open eyes. The region determination unit 113 also determines the region obtained by further extending region E6, which is a makeup region for closed eyes for deep double eyelids, in the vertical direction and adding a region corresponding to the eyes to that region, as region E3, which is a makeup region for open eyes. The function of the region determination unit 113 is realized, for example, by the function of the processor 11.

The makeup part generation unit 114 generates a first makeup part that is a makeup part to be overlaid on the first makeup area, and a second makeup part that is a makeup part to be overlaid on the second makeup area. The makeup part generation unit 114 generates the first makeup part based on the cosmetics selected by the cosmetics selection unit 106 and the first makeup area determined by the area determination unit 113. The makeup part generation unit 114 also generates the second makeup part based on the cosmetics selected by the cosmetics selection unit 106 and the second makeup area determined by the area determination unit 113. The function of the makeup part generation unit 114 is realized, for example, by the function of the processor 11.

The makeup parts holding unit 115 holds the first makeup parts and the second makeup parts generated by the makeup parts generation unit 114. The first makeup parts stored in the makeup parts holding unit 115 are superimposed on the first face image by the display control unit 103. The second makeup parts stored in the makeup parts holding unit 115 are superimposed on the second face image by the display control unit 103. The first makeup image is generated by processing the first makeup parts to be superimposed on the first makeup area in the first face image. The second makeup image is generated by processing the second makeup parts to be superimposed on the second makeup area in the second face image. The function of the makeup parts holding unit 115 is realized, for example, by the function of the flash memory 12.

Next, the image display process executed by the image processing device 10 will be described with reference to the flowchart shown in FIG.

First, the processor 11 included in the image processing device 10 displays a cosmetic image (step S101). Specifically, the processor 11 displays the cosmetic image on the second touch screen 15. When the processor 11 completes the processing of step S101, it starts capturing images (step S102). Specifically, the processor 11 causes the camera 13 to start capturing images, and thereafter, continuously acquires captured images from the camera 13.

When the processor 11 completes the process of step S102, it displays the captured image (step S103). Specifically, the processor 11 causes the captured image acquired from the camera 13 to be displayed on the first touch screen 14. When the processor 11 completes the process of step S103, it executes a type determination process (step S104). The type determination process will be described in detail with reference to FIG. 12.

First, the processor 11 determines whether the captured image is an image of eyes open (step S201). If the processor 11 determines that the captured image is not an image of eyes open (step S201: NO), the type determination process is completed. If the processor 11 determines that the captured image is an image of eyes open (step S201: YES), the processor 11 detects the coordinates of the eyes (step S202). For example, the processor 11 detects a circular image included in the captured image, and detects the position of the detected circular image as the coordinates of the eye.

When the processor 11 completes the process of step S202, it detects the lines around the eyes (step S203). For example, the processor 11 detects the 12 lines shown in FIG. 7. When the processor 11 completes the process of step S203, it determines whether or not the line L10 has been detected (step S204). If the processor 11 determines that the line L10 has not been detected (step S204: NO), it determines that the upper eyelid is a single eyelid (step S205).

When the processor 11 determines that the line L10 has been detected (step S204: YES), it identifies the gap area (step S206). When the processor 11 completes the processing of step S206, it determines whether the length of the gap area is equal to or greater than a threshold (step S207). When the processor 11 determines that the length of the gap area is equal to or greater than a threshold (step S207: YES), it determines that the upper eyelid is a double eyelid (step S208). When the processor 11 determines that the length of the gap area is not equal to or greater than the threshold (step S207: NO), it determines that the upper eyelid is a deep double eyelid (step S209).

When the processor 11 completes the processing of steps S205, S208, and S209, it stores type information (step S210). The type information is information indicating whether the upper eyelid is a single eyelid, a double eyelid, or a deep double eyelid, and is stored in the flash memory 12. When the processor 11 completes the processing of step S210, it completes the type determination process.

When the processor 11 completes the process of step S104, it determines whether or not the pointing device 30 is in contact with the second touch screen 15 (step S105). When the processor 11 determines that the pointing device 30 is in contact with the second touch screen 15 (step S105: YES), it selects a cosmetic product (step S106). When the processor 11 determines that the pointing device 30 is not in contact with the second touch screen 15 (step S105: NO), or when the processor 11 completes the process of step S106, it determines whether or not the pointing device 30 has contacted a face in the captured image (step S107).

When the processor 11 determines that the instruction device 30 has contacted the face in the captured image (step S107: YES), the processor 11 determines whether or not contact information has been received (step S108). When the processor 11 determines that contact information has been received (step S108: YES), the processor 11 executes a region determination process (step S109). The region determination process will be described in detail with reference to FIG. 13.

First, processor 11 determines the basic makeup area (step S301). Specifically, processor 11 determines the basic makeup area from the contact position. When processor 11 completes the processing of step S301, processor 11 determines whether the captured image is an image with eyes closed (step S302). If processor 11 determines that the captured image is an image with eyes closed (step S302: YES), processor 11 determines the basic makeup area to be an image with closed eyes (step S303).

When the processor 11 completes the processing of step S303, it determines whether the user's upper eyelids are single eyelids (step S304). If the processor 11 determines that the user's upper eyelids are single eyelids (step S304: YES), it determines a larger makeup area as an eye-opening makeup area (step S305). If the processor 11 determines that the user's upper eyelids are not single eyelids (step S304: NO), it determines whether the user's upper eyelids are double eyelids (step S306).

If the processor 11 determines that the user's upper eyelids are double eyelids (step S306: YES), it determines a medium makeup area as the eye-opening makeup area (step S307). If the processor 11 determines that the user's upper eyelids are not double eyelids (step S306: NO), it determines a smaller makeup area as the eye-opening makeup area (step S308).

When the processor 11 determines that the captured image is not an image of closed eyes (step S302: NO), it determines the basic makeup area to be an image of open-eyed makeup (step S309). When the processor 11 completes the processing of step S309, it determines whether the user's upper eyelids are single eyelids (step S310). When the processor 11 determines that the user's upper eyelids are single eyelids (step S310: YES), it determines a smaller makeup area as the closed-eye makeup area (step S311). When the processor 11 determines that the user's upper eyelids are not single eyelids (step S310: NO), it determines whether the user's upper eyelids are double eyelids (step S312).

If the processor 11 determines that the user's upper eyelids are double eyelids (step S312: YES), it determines a medium makeup area as the closed-eye makeup area (step S313). If the processor 11 determines that the user's upper eyelids are not double eyelids (step S312: NO), it determines a large makeup area as the closed-eye makeup area (step S314).

When the processor 11 completes the processing of steps S305, S307, S308, S311, S313, and S314, it stores the area information (step S315). When the processor 11 completes the processing of step S315, it completes the area determination processing.

When the processor 11 completes the process of step S109, it generates makeup parts for closed eyes and makeup parts for open eyes (step S110). When the processor 11 determines that the instruction device 30 is not in contact with the face in the captured image (step S107: NO), when it determines that contact information has not been received (step S108: NO), or when it completes the process of step S110, it determines whether the captured image is an image with closed eyes (step S111).

When the processor 11 determines that the captured image is an image of closed eyes (step S111: YES), it displays a closed-eyes makeup image in which closed-eyes makeup parts are superimposed on the closed-eyes makeup area (step S112). When the processor 11 determines that the captured image is not an image of closed eyes (step S111: NO), it displays an open-eyes makeup image in which open-eyes makeup parts are superimposed on the open-eyes makeup area (step S113). When the processor 11 completes the process of step S112 or step S113, it returns to step S103.

According to this embodiment, the state of the face with cosmetics applied can be appropriately simulated according to the type of facial feature. In this embodiment, even if the makeup area is the same size when applying makeup, if the state of the eyes changes, the size of the makeup area is set appropriately according to the characteristics of the eyes (type of upper eyelid), making it possible to realize a more realistic makeup simulation. In other words, in this embodiment, the makeup area for closed eyes and the makeup area for open eyes are set to appropriate sizes depending on whether the upper eyelid is single eyelid or not, so that a natural makeup image can be displayed even if the state of the user's eyes changes during the makeup simulation. Furthermore, according to this embodiment, it is possible to easily detect whether the upper eyelid is single eyelid or not by detecting the first line.

In addition, according to this embodiment, the makeup area for closed eyes and the makeup area for open eyes are set to appropriate sizes depending on whether the upper eyelid is a single eyelid, a double eyelid, or a deep double eyelid, so that even if the state of the user's eyes changes during the makeup simulation, a natural makeup image can be displayed. In addition, according to this embodiment, by detecting the first line and the second line, it is possible to easily detect whether the upper eyelid is a single eyelid, a double eyelid, or a deep double eyelid. In addition, according to this embodiment, by determining the gap area between the first line and the second line, it is possible to easily detect whether the upper eyelid is a single eyelid, a double eyelid, or a deep double eyelid. In addition, according to this embodiment, it is possible to perform a makeup simulation using an indicator.

(Embodiment 2)
In the first embodiment, the makeup simulation device 100 is described, which includes the image processing device 10 and the instruction device 30. The image processing device 10 can also function as the makeup simulation device by itself.

Hereinafter, the functions of the image processing device 10 according to this embodiment will be described with reference to FIG. 14. Functionally, the image processing device 10 according to this embodiment does not include an information receiving unit 107 and a contact position identifying unit 108, but includes an application operation receiving unit 116.

The application operation receiving unit 116 receives a simulated application operation of cosmetics by the user. This application operation is not an actual contact operation on the user's face, but an operation on the captured image displayed on the first touch screen 14. The function of the application operation receiving unit 116 is realized, for example, by the processor 11 and the first touch screen 14 working together.

The area determination unit 113 determines a first makeup area and a second makeup area based on the application operation received by the application operation reception unit 116. Specifically, the area determination unit 113 determines the first makeup area and the second makeup area based on information indicating coordinates in the captured image specified by the application operation.

According to this embodiment, the image processing device 10 alone can appropriately simulate the state of a face with cosmetics applied according to the shape of the facial features.

(Modification)
Although the embodiment of the present invention has been described above, various modifications and applications are possible in carrying out the present invention.

In the present invention, it is up to the discretion of which parts of the configurations, functions, and operations described in the above embodiments are adopted. Furthermore, in addition to the above-mentioned configurations, functions, and operations, further configurations, functions, and operations may be adopted in the present invention. Furthermore, the above-mentioned embodiments can be freely combined as appropriate. Furthermore, the number of components described in the above-mentioned embodiments can be adjusted as appropriate. Furthermore, it goes without saying that the materials, sizes, electrical characteristics, etc. that can be adopted in the present invention are not limited to those shown in the above-mentioned embodiments.

In the above embodiment, an example has been described in which the CPU in the processor 11 executes a program stored in the ROM to execute various processes. In the present invention, instead of the processor 11, dedicated hardware such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or various control circuits may be provided, and the dedicated hardware may execute various processes. In this case, the functions of each part may be realized by individual hardware, or the functions of each part may be realized together by a single piece of hardware. Also, some of the functions of each part may be realized by dedicated hardware, and the other parts may be realized by software or firmware.

In addition to being able to provide an image processing device that is already equipped with the configuration for realizing the functions of the present invention, by applying a program, an existing information processing device or the like can be made to function as an image processing device of the present invention. In other words, by applying a program for realizing each functional configuration of the image processing device 10 exemplified in the above embodiment so that it can be executed by a CPU or the like that controls an existing information processing device, it can be made to function as an image processing device of the present invention.

The method of applying such a program is arbitrary. The program can be applied by storing it on a computer-readable storage medium, such as a flexible disk, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, or a memory card. Furthermore, the program can be applied via a communication medium such as the Internet by superimposing it on a carrier wave. For example, the program can be distributed by posting it on a bulletin board system (BBS) on a communication network. The program can then be started and executed under the control of an OS (Operating System) in the same way as other application programs, thereby enabling the above processing to be performed.

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the specific embodiment, and the present invention includes the inventions described in the claims and their equivalents. The inventions described in the original claims of this application are listed below.
(Appendix 1)
an image acquisition means for acquiring a first face image, which is a face image of a user captured when a face part is in a first state, and a second face image, which is a face image of the user captured when the face part is in a second state different from the first state;
a display means for displaying the face image of the user acquired by the image acquisition means;
a type determining means for determining a type of the facial feature of the user based on at least one of the first face image and the second face image acquired by the image acquiring means;
a region determining means for determining a first makeup area on the first face image based on a makeup operation performed by the user when the facial feature is in the first state, and for determining a second makeup area, the second makeup area having a size different from that of the first makeup area, on the second face image based on the makeup operation and the type of the facial feature determined by the type determining means.
Image processing device.
(Appendix 2)
a display control means for causing the display means to display a first makeup image in which the first makeup area on the first facial image has been processed when the image acquisition means acquires the first facial image after the makeup application is performed, and for causing the display means to display a second makeup image in which the second makeup area on the second facial image has been processed when the image acquisition means acquires the second facial image after the makeup application is performed,
2. The image processing device according to claim 1.
(Appendix 3)
the first makeup image is an image in which makeup parts expressing a state in which the cosmetic product designated by the makeup action is applied to the face of the user are superimposed on the first makeup area on the first face image,
the second makeup image is an image in which the makeup parts are superimposed on the second makeup area on the second face image;
3. The image processing device according to claim 2.
(Appendix 4)
The facial feature is an upper eyelid,
the first state being a closed state and the second state being an open state;
The type determination means determines whether the upper eyelid of the user is a single eyelid,
the area determination means determines an area having a first size as the second makeup area when the type determination means determines that the upper eyelid of the user is a single eyelid, and determines an area having a size smaller than the first size as the second makeup area when the type determination means determines that the upper eyelid of the user is not a single eyelid.
4. The image processing device according to claim 1 .
(Appendix 5)
The type determination means is
a line detection means for detecting a first line representing a groove of a double eyelid or a deep double eyelid from the second face image acquired by the image acquisition means;
and a line discrimination means for discriminating that the upper eyelid of the user is a single eyelid when the first line is not detected by the line detection means, and for discriminating that the upper eyelid of the user is not a single eyelid when the first line is detected by the line detection means.
5. The image processing device according to claim 4.
(Appendix 6)
The type discrimination means discriminates whether the upper eyelid of the user is a single eyelid, a double eyelid, or a deep double eyelid,
The area determination means determines an area having the first size as the second makeup area when the type determination means determines that the upper eyelid of the user is a single eyelid, determines an area having a second size smaller than the first size as the second makeup area when the type determination means determines that the upper eyelid of the user is a double eyelid, and determines an area having a third size smaller than the second size as the second makeup area when the type determination means determines that the upper eyelid of the user is a deep double eyelid.
5. The image processing device according to claim 4.
(Appendix 7)
The type determination means is
a line detection means for detecting a first line representing a sulcus of a double eyelid or a deep double eyelid and a second line representing a mucous membrane portion at the boundary between the upper eyelid and the eye from the second face image acquired by the image acquisition means;
and a line discrimination means for discriminating whether the upper eyelid of the user is a single eyelid, a double eyelid, or a deep double eyelid based on the first line and the second line detected by the line detection means.
7. The image processing device according to claim 6.
(Appendix 8)
The line discrimination means discriminates that the upper eyelid of the user is a single eyelid when the first line is not detected by the line detection means, discriminates that the upper eyelid of the user is a double eyelid when the length in the width direction of the face of the gap area between the first line and the second line is equal to or greater than a threshold, and discriminates that the upper eyelid of the user is a deep double eyelid when the length is less than the threshold.
8. The image processing device according to claim 7.
(Appendix 9)
the display control means causes the display means to display the first face image or the second face image in a first display area, and to display a cosmetic image representing a cosmetic product in a second display area different from the first display area;
an operation receiving means for receiving a user operation involving contact with the second display area using a pointer while the cosmetic image is being displayed in the second display area by the display means;
a contact position specifying means for specifying, when the indicator comes into contact with the face of the user while the display means is displaying the first face image after the operation accepting means has accepted the user operation, a contact position on the user's face where the indicator has come into contact based on the first face image,
the area determining means determines an area on the first face image corresponding to the contact position identified by the contact position identifying means as the first makeup area, and determines an area on the second face image corresponding to the contact position and a shape of the facial feature as the second makeup area;
the first makeup image is an image in which makeup parts expressing a state in which the cosmetic product is applied to the face of the user are superimposed on the first makeup area on the first face image,
the second makeup image is an image in which the makeup parts are superimposed on the second makeup area on the second face image;
3. The image processing device according to claim 2.
(Appendix 10)
acquiring a first face image, which is a face image of a user captured when a face part is in a first state, and a second face image, which is a face image of the user captured when the face part is in a second state different from the first state;
Displaying a facial image of the user;
determining a type of the facial feature of the user based on at least one of the first face image and the second face image;
determining a first makeup area on the first face image based on a makeup operation performed by the user when the facial feature is in the first state, and determining a second makeup area on the second face image, the second makeup area having a size different from that of the first makeup area, based on the makeup operation and a shape of the facial feature;
Image processing methods.
(Appendix 11)
Computer,
an image acquisition means for acquiring a first face image, which is a face image of a user captured when a face part is in a first state, and a second face image, which is a face image of the user captured when the face part is in a second state different from the first state;
a display control means for displaying the face image of the user acquired by the image acquisition means on a display device;
a type determining means for determining a type of the facial feature of the user based on at least one of the first face image and the second face image acquired by the image acquiring means;
a region determining means for determining a first makeup area on the first face image based on a makeup application performed by the user when the facial feature is in the first state, and for determining a second makeup area on the second face image, the second makeup area having a size different from that of the first makeup area, based on the makeup application and the type of the facial feature determined by the type determining means;
program.

10...image processing device, 11...processor, 12...flash memory, 13...camera, 14...first touch screen, 15...second touch screen, 16...communication interface, 30...instruction device, 31...processor, 32...flash memory, 33...pressure sensor, 34...communication interface, 100...makeup simulation device, 101...imaging unit, 102...image acquisition unit, 103...display control unit, 104...display unit, 105...operation reception unit, 106...cosmetic selection unit, 107...information Information receiving unit, 108...contact position identification unit, 109...state discrimination unit, 110...type discrimination unit, 111...line detection unit, 112...line discrimination unit, 113...area determination unit, 114...makeup part generation unit, 115...makeup part holding unit, 116...application operation reception unit, 301...contact detection unit, 302...information transmission unit, L1, L2, L3, L4, L5, L6, L6A, L6B, L6C, L7, L8, L9, L10, L10B, L10C, L11, L12...lines, E1, E2, E3, E4, E5, E6...areas

Claims (4)

an image acquisition means for acquiring an open-eye image, which is a facial image of a user captured when the upper eyelid is open;
and an area determining means for determining a makeup area on the eye-open image of the user based on a makeup action performed by the user when the upper eyelid is closed and a type of the upper eyelid, so that the makeup area when the upper eyelid of the user is a single eyelid is larger than the makeup area when the upper eyelid of the user is a deep double eyelid.
Image processing device. A type discrimination means for discriminating a type of the upper eyelid of the user based on the eye-open image acquired by the image acquisition means,
The area determining means determines the makeup area based on the makeup operation and the type of the upper eyelid determined by the type determining means.
The image processing device according to claim 1 . An open-eye image is obtained, which is a facial image of the user captured when the upper eyelid is open;
When determining a makeup area on the eye-open image of the user based on the makeup action performed by the user when the upper eyelid is closed and the type of the upper eyelid, the makeup area is determined so that a makeup area when the upper eyelid of the user is a single eyelid is larger than a makeup area when the upper eyelid of the user is a deep double eyelid.
Image processing methods. Computer,
An image acquisition means for acquiring an open-eye image, which is a facial image of the user captured when the upper eyelid is open;
and functioning as an area determining means for determining, when determining a makeup area on the eye-open image of the user based on the makeup action performed by the user when the upper eyelid is closed and the type of the upper eyelid, a makeup area when the upper eyelid of the user is a single eyelid is larger than a makeup area when the upper eyelid of the user is a deep double eyelid.
program.

Images