JP7704288B2 - Image conversion device, image conversion method, and image conversion program - Google Patents

Description

Embodiments of the present invention relate to an image conversion device, an image conversion method, and an image conversion program.

Non-Patent Document 1 discloses the possibility of manipulating emotional experiences by real-time facial deformation feedback. In Non-Patent Document 1, the subject's face is tracked in real time and a natural facial deformation process is performed. In Non-Patent Document 1, the Rigid MLS (Moving Least Squares) method is used as an image conversion method to deform the facial expression in the facial image. The Rigid MLS method is a technique in which feature points in an image recognized from an image are used as control points, and the image is distorted by moving each control point. Note that the facial image may be an image of the subject's face, an image of an avatar's face extracted by a computer, etc.

Shigeaki Yoshida et al., "Manipulation of Emotional Experience by Real-time Facial Deformation Feedback," Journal of Human Interface Society, Vol. 17, No. 1, 2015

When recognizing facial parts, there are feature point recognition methods that only recognize one side of the facial part. If you try to move a facial part recognized by such a feature point recognition method using the Rigid MLS method, you will only get an image of a face with an unnatural expression.

For example, there is a feature point recognition method that recognizes only one side of eyebrows, the upper side, and not the other lower side. If only the upper side of eyebrows is recognized using this method, and the image is transformed to move them upwards, the resulting facial image will have thicker eyebrows and an unnatural facial expression. In addition, if only one side of the width of the double eyelid, the shading of the contour, etc. is recognized, the facial image will also have an unnatural facial expression when the image is transformed.

This invention aims to provide an image conversion technology that can convert an image of a facial expression into one that is natural, even when only one side of the facial feature is recognized.

In order to solve the above problem, an image conversion device according to one aspect of the present invention includes a control point generation unit and a facial expression conversion unit. The control point generation unit adds a second feature point, which is a feature point on the other side that has not been recognized, based on a first feature point, which is a feature point on one side of a facial feature recognized from an image of a human face, and sets the first and second feature points as control points. The facial expression conversion unit obtains a converted image in which the facial expression of the human face has been converted by transforming the control point by an amount of transformation corresponding to the conversion facial expression to be converted.

According to one aspect of the invention, an image is converted by adding feature points on one side of a facial feature based on feature points on the other side, thereby providing an image conversion technology that can convert a facial image into one with a natural expression even when only one side of a facial feature is recognized.

FIG. 1 is a block diagram showing an example of the configuration of an image conversion device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a hardware configuration of the image conversion device. FIG. 3 is a diagram showing an example of facial feature points. FIG. 4 is a diagram showing an example of a storage format of feature points. FIG. 5 is a diagram showing an example of a storage format of the amount of change. FIG. 6 is a flowchart showing an example of an image conversion processing operation by the image conversion device. FIG. 7 is a schematic diagram for explaining the relationship between the eyebrow feature points and the feature points above the eyes. FIG. 8 is a schematic diagram for explaining a method of adding feature points below the eyebrows. FIG. 9 is a schematic diagram for explaining a method of adding eyelid feature points.

[One embodiment]
An embodiment of the present invention will now be described with reference to the drawings.

(Configuration example)
1 is a block diagram showing an example of the configuration of an image conversion device 1 according to an embodiment of the present invention. The image conversion device 1 has an image acquisition unit 11, a feature point recognition unit 12, a control point generation unit 13, a converted facial expression input unit 14, a change amount storage unit 15, a facial expression conversion unit 16, and an image output unit 17.

The image acquisition unit 11 acquires a facial image from a web camera, an avatar, etc. The image acquisition unit 11 outputs the acquired facial image to the feature point recognition unit 12 and the facial expression conversion unit 16.

The feature point recognition unit 12 receives the facial image acquired by the image acquisition unit 11 as input and recognizes feature points from the facial image. The method for recognizing feature points in the feature point recognition unit 12 will be described later. The feature point recognition unit 12 outputs the recognized feature points to the control point generation unit 13.

The control point generation unit 13 receives the first feature points recognized by the feature point recognition unit 12 as input, and adds the second feature points that are not recognized based on the input first feature points. For example, the control point generation unit 13 calculates the distance between the eyebrow feature points and the eye feature points that are the first feature points, and adds the second feature points half the calculated distance below each eyebrow feature point. The method of adding the second feature points will be described in detail later. The control point generation unit 13 outputs the first and second feature points as control points to the facial expression conversion unit 16. It is predetermined which of the first feature points the second feature points are added based on, and the number of second feature points to be added. Therefore, the number of control points is also predetermined.

The conversion facial expression input unit 14 acquires a conversion facial expression, which is the facial expression to be converted to, such as a smile, that is specified and input by the user from a user interface such as a keyboard. The conversion facial expression input unit 14 outputs the acquired conversion facial expression to the facial expression conversion unit 16.

The change amount storage unit 15 pre-stores the change amount for each control point for each facial expression to be converted. The change amount is information indicating how far the control point should be moved. The change amount can be determined in advance, for example, by a user applying facial expression transformation processing to an expressionless face for a specific facial image, adjusting it to produce a natural expression.

The facial expression conversion unit 16 receives as input the facial image acquired by the image acquisition unit 11, the control points output by the control point generation unit 13, and the converted facial expression acquired by the converted facial expression input unit 14. The facial expression conversion unit 16 also reads out from the change amount storage unit 15 the amount of change in the facial expression to be converted, which is indicated by the converted facial expression input from the converted facial expression input unit 14. The change amount storage unit 15 moves each control point in the input facial image based on the amount of movement of the read control points, thereby obtaining a facial image in which the facial expression of the facial image has been converted. The facial expression conversion unit 16 outputs the converted facial image to the image output unit 17.

The image output unit 17 receives the converted facial image from the facial expression conversion unit 16 as input, and outputs the input facial image. Here, output includes, for example, storing the image in a storage medium, displaying the image on a display, transmitting the image to another device via a communication network, and the like.

Figure 2 is a diagram showing an example of the hardware configuration of the image conversion device 1.

The image conversion device 1 is composed of a computer such as a personal computer, a smartphone, or a server computer. As shown in FIG. 2, the image conversion device 1 has a hardware processor 100 such as a CPU (Central Processing Unit). The CPU can execute multiple information processes simultaneously by using a multi-core and multi-threaded one. The processor 100 may also have multiple CPUs. In the image conversion device 1, a program memory 200, a data memory 300, a communication interface 400, and an input/output interface (referred to as input/output IF in FIG. 2) 500 are connected to the processor 100 via a bus 600.

The communication interface 400 may include, for example, one or more wired or wireless communication modules. The communication interface 400 may communicate with other computers, web cameras, etc. that are connected via a cable or a network such as a LAN (Local Area Network) or the Internet.

The input unit 700 and the display unit 800 are connected to the input/output interface 500. The input unit 700 includes input devices such as a keyboard, a pointing device such as a mouse, and a sensor device such as a camera. The display unit 800 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display. The input unit 700 and the display unit 800 can also use a so-called tablet-type input/display device. This type of input/display device is configured by arranging an input detection sheet using an electrostatic method or a pressure method on the display screen of a display device using, for example, liquid crystal or organic EL (Electro Luminescence). The input/output interface 500 inputs operation information inputted in the input unit 700 to the processor 100 and causes the display unit 800 to display display information generated by the processor 100.

In addition, the input unit 700 and the display unit 800 do not have to be connected to the input/output interface 500. The input unit 700 and the display unit 800 can transmit and receive information between the processor 100 and the input unit 700 and the display unit 800 by being provided with a communication unit for connecting to the communication interface 400 directly or via a network.

In addition, the input/output interface 500 may have a function for reading/writing a recording medium such as a semiconductor memory such as a flash memory, or may have a function for connecting to a reader/writer having a function for reading/writing such a recording medium. Furthermore, the input/output interface 500 may have a function for connecting to other devices.

The program memory 200 is a non-transient tangible computer-readable storage medium that is a combination of a non-volatile memory that can be written and read at any time and a non-volatile memory that can only be read at any time. Examples of non-volatile memories that can be written and read at any time include HDDs (Hard Disk Drives) and SSDs (Solid State Drives). Examples of non-volatile memories that can only be read at any time include ROMs. The program memory 200 stores programs, such as an image conversion program, that are necessary for the processor 100 to execute various control processes according to one embodiment. That is, the processing function units in each of the image acquisition unit 11, feature point recognition unit 12, control point generation unit 13, conversion facial expression input unit 14, change amount storage unit 15, facial expression conversion unit 16, and image output unit 17 can all be realized by having the processor 100 read and execute the image conversion program stored in the program memory 200. It should be noted that some or all of these processing functions may be implemented in a variety of other forms, including integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

The data memory 300 is a tangible computer-readable storage medium, for example, a combination of the above-mentioned non-volatile memory and a volatile memory such as a RAM (Random Access Memory). This data memory 300 is used to store various data acquired and created in the process of performing various processes. That is, in the data memory 300, an area is secured for storing various data as appropriate in the process of performing various processes. As such areas, the data memory 300 may be provided with, for example, an acquired image storage unit 301, a feature point storage unit 302, a control point storage unit 303, a converted facial expression designation storage unit 304, a change amount storage unit 305, a converted image storage unit 306, and a temporary storage unit 307.

The acquired image memory unit 301 is used to store facial images acquired when the processor 100 operates as the above-mentioned image acquisition unit 11.

The feature point memory unit 302 is used to store feature points acquired when the processor 100 operates as the above-mentioned feature point recognition unit 12.

Figure 3 is a diagram showing an example of facial feature points. The stars in Figure 3 are feature points recognized by the processor 100, and the numbers next to each feature point are unique feature point IDs for identifying each feature point. The number of feature point IDs and the part of the face that each feature point ID corresponds to are determined by the feature point recognition method employed. For example, the feature point for feature point ID "18" is predetermined to be the left end of the left eyebrow as viewed from the front.

Figure 4 is a diagram showing an example of the storage format of feature points in the feature point storage unit 302. As shown in Figure 4, the feature point storage unit 302 stores the x and y coordinates in the face image in a table format, corresponding to the feature point ID. The coordinate values are in pixels. Therefore, in the example of Figure 3, the feature point storage unit 302 stores the x and y coordinates for feature points with feature point IDs "1" to "68".

The control point storage unit 303 is used to store the control points generated when the processor 100 operates as the control point generation unit 13. The storage format of the control points in the control point storage unit 303 is, for example, the same as the storage format of the feature points in the feature point storage unit 302 shown in FIG. 4. That is, the control point storage unit 303 can store the x and y coordinates in the face image in a table format in association with the control point ID. The control point storage unit 303 stores the feature point IDs "1" to "68" assigned to the feature points shown in FIG. 3 as control point IDs "1" to "68" in association with the x and y coordinates of the feature point IDs "1" to "68". The processor 100 also stores the x and y coordinates of the second feature point, which is the added feature point, in association with the control point IDs "69" and onward.

The conversion facial expression specification memory unit 304 is used to store the conversion facial expression specified by the user, which is obtained when the processor 100 operates as the above-mentioned conversion facial expression input unit 14.

The change amount memory unit 305 corresponds to the change amount storage unit 15 described above.

Figure 5 is a diagram showing an example of the storage format of the amount of change in the change amount storage unit 305. As shown in Figure 5, the change amount storage unit 305 can be in a table format that stores the amount of change in the x coordinate and the amount of change in the y coordinate for each converted facial expression in association with a control point ID. The value of the amount of change is in pixels. The amount of change is represented by the direction and amount of movement of the control point. For example, a movement amount of "+1" represents a movement of one pixel in the positive direction.

The converted image memory unit 306 is used to store the converted facial image when the processor 100 operates as the above-mentioned facial expression conversion unit 16.

The temporary memory unit 307 is used to store various intermediate data that is generated during operation of the processor 100 and is not stored in the acquired image memory unit 301, the feature point memory unit 302, the control point memory unit 303, the converted facial expression designation memory unit 304, the change amount memory unit 305 and the converted image memory unit 306.

(operation)
Next, the operation of the image conversion device 1 having the image conversion device 1 will be described.

Figure 6 is a flowchart showing an example of an image conversion processing operation by image conversion device 1. Processor 100 of image conversion device 1 starts operation as image conversion device 1 shown in this flowchart by reading and executing an image conversion program stored in program memory 200. Execution of the image conversion program by processor 100 is started when an instruction to perform image conversion is received from input unit 700 via input/output interface 500 or via communication interface 400.

The processor 100 operates as the conversion facial expression input unit 14 and waits for the user to input a designated conversion facial expression, such as a smile, which is the facial expression to be converted to (step S1). For example, the processor 100 determines whether the input signal from the input unit 700 via the input/output interface 500 or the communication interface 400 includes a designated input of a conversion facial expression. If a designated input of a conversion facial expression is present, the processor 100 proceeds to processing of step S2.

The processor 100 stores the specified converted facial expression in the converted facial expression specification memory unit 304 of the data memory 300 (step S2).

The processor 100 operates as the image acquisition unit 11 to acquire a facial image (step S3). For example, the processor 100 acquires an image of the subject's face captured by the camera of the input unit 700 via the input/output interface 500. Alternatively, the processor 100 acquires a facial image captured by a web camera connected to a network or the face of an avatar generated by another computer via the communication interface 400. The processor 100 stores the acquired facial image in the acquired image storage unit 301 of the data memory 300.

The processor 100 operates as the feature point recognition unit 12 and recognizes the first feature point from the face image stored in the acquired image storage unit 301 (step S4). The processor 100 recognizes feature points from the face image, for example, by using the face_landmark_detection function of dlib (see, for example, http://dlib.net/face_landmark_detection.py.html). Specifically, the processor 100 extracts a distribution of the gradient direction of brightness, called a HOG (Histogram of Oriented Gradients) feature, from the input face image. A model that is trained based on data that links the HOG feature to the positions of the facial feature points is generally provided. Therefore, the processor 100 inputs the extracted HOG feature into this learning model and acquires the positions of the facial feature points. The processor 100 stores the positions of the acquired first feature points in the feature point storage unit 302 of the data memory 300.

The processor 100 operates as the control point generator 13 to generate control points (step S5). Specifically, the processor 100 stores the recognized first feature points as control points in the control point storage unit 303 of the data memory 300. The processor 100 further adds second feature points, which are feature points on the other side, to face parts for which feature points are only recognized on one side. The processor 100 then stores the added second feature points as additional control points in the control point storage unit 303.

Examples of facial features for which feature points are only recognized on one side include eyebrows, eyelids, and contours. As only the upper feature points of the eyebrows are recognized, processor 100 adds feature points below the eyebrows. As only the upper feature points of the eyelids, which are the lower feature points of the double eyelids, are recognized, processor 100 adds feature points above them. As the contours are recognized as feature points in the non-shaded parts, processor 100 adds feature points in the shaded parts.

For example, the processor 100 adds eyebrow feature points as follows.
7 is a schematic diagram for explaining the relationship between the feature points of the eyebrows and the feature points above the eyes. The processor 100 draws a perpendicular line from each feature point of the eyebrows (feature points with feature point IDs "18" to "22") and obtains the feature point above the eyes (feature points with feature point IDs "37" to "40") that is closest to the perpendicular line. That is, the processor 100 obtains the feature point ID "37" for the feature point ID "18", the feature point ID "37" for the feature point ID "19", the feature point ID "38" for the feature point ID "20", the feature point ID "39" for the feature point ID "21", the feature point ID "40" for the feature point ID "22", ... and the feature point ID "46" for the feature point ID "27". At this time, d18, d19, d20, d21, d22, . . . , d27 are obtained as the distances (differences in ordinates) between the respective feature points.

Figure 8 is a schematic diagram for explaining a method for adding a lower feature point, which is a second feature point of an eyebrow. The processor 100 calculates the average distance da of the distances d18 to d27 between the above feature points. At this time, the average distance da may be calculated without distinguishing between the right eye and the left eye, or, since there is generally a slight difference between the left and right eyes, the average distance da may be calculated separately. Here, the average distance is calculated by distinguishing between the left and right. That is, the processor 100 calculates the average distance da for the left eyebrow as follows: da = (d18 + d19 + d20 + d21 + d22)/5, and for the right eyebrow as follows: da = (d23 + d24 + d25 + d26 + d27)/5.

The processor 100 sets the feature point addition distance d to half the average distance da thus calculated, i.e., da/2, and adds second feature points the feature point addition distance d below each of the first feature points of the eyebrows. That is, the processor 100 adds a second feature point with feature point ID "69" the feature point addition distance d below the first feature point with feature point ID "18". In the same manner, the processor 100 adds a second feature point with feature point ID "70" below the first feature point with feature point ID "19", a second feature point with feature point ID "71" below the first feature point with feature point ID "20", ..., a second feature point with feature point ID "78" below the first feature point with feature point ID "27".

The processor 100 also adds eyelid feature points, for example, in the following manner.
9 is a schematic diagram for explaining a method for adding a second feature point of an eyelid. The feature point on one side of the recognized eyelid is the feature point on the upper side of the eye. Therefore, when adding a second feature point of the eyelid, the average distance da used when adding the second feature point of the eyebrow can be used. The processor 100 sets the feature point adding distance d to be 1/4 of the average distance da for the left and right, that is, da/4. The processor 100 adds a second feature point at a distance d above each of the feature points above the eyes (first feature points with feature point IDs "37" to "40"). That is, the processor 100 adds a second feature point with feature point ID "79" at a distance d above the first feature point with feature point ID "37". In a similar manner, the processor 100 adds a second feature point with feature point ID "80" on top of the first feature point with feature point ID "38", a second feature point with feature point ID "81" on top of the first feature point with feature point ID "39", ..., a second feature point with feature point ID "86" on top of the first feature point with feature point ID "46".

As there is not much individual variation in the size of the contour shading, the processor 100 adds a second feature point to the contour shading, for example, as follows. For example, the processor 100 adds a second feature point to each of the contour feature points (first feature points with feature point IDs "1" to "17") at a position in a predetermined direction and distance.

The processor 100 operates as the facial expression conversion unit 16 to convert the facial expression of the facial image stored in the acquired image storage unit 301 (step S6). That is, the processor 100 converts the facial image based on the control points stored in the control point storage unit 303 and the change amount stored in the change amount storage unit 305 according to the converted facial expression stored in the converted facial expression designation storage unit 304. For example, the processor 100 uses an implementation of MLS (see, for example, https://github.com/Jarvis73/Moving-Least-Squares) or the like.
Specifically, processor 100 moves each control point by an amount of change corresponding to the converted facial expression stored in converted facial expression designation storage unit 304. For example, when changing the facial expression to a smile, since the x and y coordinates before conversion for control point ID "1" are (23, 45) (see FIG. 4), processor 100 performs a conversion such that the pixel of the control point moves to (24, 47) by incrementing the x coordinate by "+1" and the y coordinate by "+2" (see FIG. 5).

For points other than the control points, the processor 100 applies the following affine transformation (including the Helmert transformation = similarity transformation and rigid deformation):

Here, x and y are the coordinates of the nearby control point, x' and y' are the coordinates obtained by adding the change amount to the coordinates of the control point, a, b, c, d are parameters, and t _x and t _y are translation parameters. The processor 100 calculates the least square mean of the coordinates x' and y' obtained by adding the change amount to the coordinates of the control point x and y, and obtains parameters a, b, c, d, t _x and t _y that minimize this by global optimization. Then, the coordinates of the target point to be transformed are set to x and y, and the transformed coordinates are obtained using these obtained parameters. The processor 100 uses the parameters a, b, c, d, t _x and t _y obtained in this way to obtain the transformed coordinates from the added control point by the above-mentioned affine transformation.

The processor 100 stores the facial image thus converted as a converted image in the converted image storage unit 306 of the data memory 300.

The processor 100 operates as the image output unit 17 and outputs the converted image stored in the converted image storage unit 306 (step S7). For example, the processor 100 causes the display unit 800 to display the facial image via the input/output interface 500. Alternatively, the processor 100 transmits the facial image onto the network via the communication interface 400 and causes it to be displayed on a display device connected to the network, or on the display unit of another computer connected to the network.

The processor 100 determines whether or not to end the operation of the image conversion device 1 shown in this flowchart (step S8). For example, the processor 100 checks whether or not the user has instructed the image conversion to end from the input unit 700, via the input/output interface 500, or via the communication interface 400. If the operation is to end, the processor 100 ends the operation shown in this flowchart.

On the other hand, if the operation has not yet ended, the processor 100 operates as the conversion facial expression input unit 14 and determines whether or not the user has input a specification to change the conversion facial expression (step S9). If there has been no input specifying a change to the conversion facial expression, the processor 100 proceeds to processing of step S3. If there has been input specifying a change to the conversion facial expression, the processor 100 proceeds to processing of step S2.

The image conversion device 1 according to the embodiment described above includes a control point generation unit 13 and a facial expression conversion unit 16. Based on a first feature point that is a feature point on one side of a facial feature recognized from an image of a person's face, the control point generation unit 13 adds a second feature point that is a feature point on the other side that is not recognized, and sets the first and second feature points as control points. The facial expression conversion unit 16 obtains a converted image in which the facial expression of the person is converted by transforming the control point by an amount of transformation corresponding to the conversion facial expression to be converted.
Therefore, the image conversion device 1 according to one embodiment converts an image by adding feature points on one side of a facial feature based on feature points on the other side, and therefore can provide an image conversion technology that can convert a facial image into one with a natural expression even when only one side of a facial feature is recognized.

Furthermore, in the image conversion device 1 according to an embodiment, the facial features include at least the eyebrows or the eyelids.
In this way, even if only feature points above the eyebrows and/or feature points below the eyelids can be recognized, it is possible to add feature points below the eyebrows and/or feature points above the eyelids and convert the face image into one with a natural expression.

Here, the control point generation unit 13 calculates a feature point addition distance d based on the distance between the feature point above the eyebrow, which is the first feature point, and the feature point above the eye, and adds a point below the eyebrow that is away from the first feature point above the eyebrow by the feature point addition distance d as a second feature point.
Therefore, a second feature point below the unrecognized eyebrow can be easily added.

Alternatively, the control point generation unit 13 calculates a feature point adding distance d based on the distance between the upper feature point of the eyebrow and the lower feature point of the eyelid, which is the first feature point, and adds a point on the upper eyelid that is the feature point adding distance d away from the first feature point of the lower eyelid, which is the first feature point, as the second feature point.
Therefore, a second feature point on the upper side of the eyelid that is not recognized can be easily added.

Moreover, the image conversion device 1 according to one embodiment includes a change amount storage unit 15 that stores in advance a change amount representing the amount of deformation for each control point for each conversion facial expression to be converted, and a conversion facial expression input unit 14 that inputs the conversion facial expression to be converted. The facial expression conversion unit 16 reads out the change amount corresponding to the input conversion facial expression from the change amount storage unit, and obtains a conversion image using the read out change amount.
In this way, by storing the change amount in advance for the control point corresponding to the second feature point to be added, it is possible to convert the face image into one with a natural expression by using the control point corresponding to the second feature point as well.

[Other embodiments]
It should be noted that the present invention is not limited to the above embodiment.
For example, the flow of each process described above is not limited to the procedures described, and the order of some steps may be changed, or some steps may be performed simultaneously in parallel.

In addition, the process flow described above is for converting facial expressions in facial images acquired in real time in real time, but it can also be applied to applications where the facial expressions of stored facial images are converted rather than real-time processing.

When adding a second feature point, a fixed value of 1/2 or 1/4 is used for the average distance da, but it may also be possible to allow the user to specify any value.

The user may also be able to select which facial features to add a second feature point, which is a feature point on the other side.

The method described in the embodiment can be stored as a program (software means) that can be executed by a calculator (computer) on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a setting program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program recorded on the recording medium, and in some cases, constructs the software means using the setting program, and executes the above-mentioned processing by controlling the operation of this software means. Note that the recording medium referred to in this specification is not limited to a storage medium for distribution, but also includes storage media such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

In short, this invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the gist of the invention. Furthermore, the embodiments may be implemented in appropriate combinations as far as possible, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriate combinations of the multiple constituent elements disclosed.

1...Image conversion device 11...Image acquisition unit 12...Feature point recognition unit 13...Control point generation unit 14...Transformed facial expression input unit 15...Change amount storage unit 16...Facial expression conversion unit 17...Image output unit 100...Processor 200...Program memory 300...Data memory 301...Acquired image storage unit 302...Feature point storage unit 303...Control point storage unit 304...Transformed facial expression designation storage unit 305...Change amount storage unit 306...Transformed image storage unit 307...Temporary storage unit 400...Communication interface 500...Input/output interface 600...Bus 700...Input unit 800...Display unit

Claims (6)

a control point generation unit that adds a second feature point that is a feature point on the other side of a face feature that has not been recognized based on a first feature point that is a feature point on one side of a face feature recognized from an image of a human face, and sets the first and second feature points as control points;
an expression conversion unit for converting the control points by a transformation amount corresponding to a transformation expression to be converted, thereby obtaining a transformation image in which the facial expression of the person is transformed;
Equipped with
The facial features include at least eyebrows or eyelids,
The control point generation unit
Calculating a feature point addition distance based on a distance between the feature point above the eyebrow and the feature point above the eye, which are the first feature points;
an image conversion device that adds, as the second feature point, a point below the eyebrow that is away from the first feature point above the eyebrow by the feature point addition distance. a control point generation unit that adds a second feature point that is a feature point on the other side of a face feature that has not been recognized based on a first feature point that is a feature point on one side of a face feature recognized from an image of a human face, and sets the first and second feature points as control points;
an expression conversion unit for converting the control points by a transformation amount corresponding to a transformation expression to be converted, thereby obtaining a transformation image in which the facial expression of the person is transformed;
Equipped with
The facial features include at least eyebrows or eyelids,
The control point generation unit
Calculating a feature point addition distance based on a distance between an upper feature point of an eyebrow and a lower feature point of an eyelid, which is the first feature point;
an image conversion device that adds, as the second feature point, a point on an upper side of an eyelid that is away from the first feature point, which is the feature point on the lower side of the eyelid, by the feature point addition distance. a change amount storage unit that stores in advance a change amount representing the deformation amount for each of the control points for each of the facial expressions to be converted;
a conversion expression input unit for inputting the conversion expression to be converted;
Further comprising:
the facial expression conversion unit reads out the amount of change corresponding to the input converted facial expression from the amount of change storage unit, and obtains the converted image using the amount of change read out.
3. An image conversion device according to claim 1 or 2 . 1. An image conversion method for an image conversion device having a processor and converting facial expressions in an image of a human face, comprising:
adding, by the processor, a second feature point which is a feature point on one side of a facial feature that has not been recognized, based on a first feature point which is a feature point on the other side of a facial feature recognized from an image of the person's face, and setting the first and second feature points as control points;
The facial features include at least eyebrows or eyelids,
Calculating a feature point addition distance based on a distance between the feature point above the eyebrow and the feature point above the eye, which are the first feature points;
adding a point below the eyebrow that is separated by the feature point addition distance from the feature point above the eyebrow, which is the first feature point, as the second feature point;
a transformation image in which the facial expression of the person is transformed by the processor by a transformation amount corresponding to the transformation facial expression to be transformed;
Image conversion methods. 1. An image conversion method for an image conversion device having a processor and converting facial expressions in an image of a human face, comprising:
adding, by the processor, a second feature point which is a feature point on one side of a facial feature that has not been recognized, based on a first feature point which is a feature point on the other side of a facial feature recognized from an image of the person's face, and setting the first and second feature points as control points;
The facial features include at least eyebrows or eyelids,
Calculating a feature point addition distance based on a distance between an upper feature point of an eyebrow and a lower feature point of an eyelid, which is the first feature point;
adding, as the second feature point, a point on an upper side of an eyelid that is separated by the feature point addition distance from the feature point on the lower side of the eyelid, which is the first feature point;
a transformation image in which the facial expression of the person is transformed by the processor by a transformation amount corresponding to the transformation facial expression to be transformed;
Image conversion methods. 4. An image conversion program for causing a processor to function as each of the units of the image conversion device according to claim 1 .

Images