JP7700951B2 - Image conversion device, method and program - Google Patents

Description

Embodiments of the present invention relate to image conversion devices, methods and programs.

Non-Patent Document 1 discloses the possibility of manipulating emotional experiences by feedback of real-time facial deformation (facial expression transformation). In Non-Patent Document 1, the subject's face is tracked in real time and a natural facial expression transformation process is performed. In Non-Patent Document 1, the Rigid MLS (Moving Least Squares) method is used as an image transformation method to deform the facial expression in the facial image. The Rigid MLS method is a method of distorting an image by recognizing and moving feature points in an image recognized from the image. Such a method is also disclosed in Non-Patent Document 2. Note that the facial image may be an image of the subject's face, an image of an avatar's face extracted by a computer, or the like.

Shigeo Yoshida et al., "Manipulation of Emotional Experience by Real-time Deformed Facial Feedback," The Transactions of Human Interface Society, Vol. 17, No. 1, 2015 Tomas Jakab, et al., “Unsupervised Learning of Object Landmarks through Conditional Image Generation”, NIPS, 2018.

However, if the angle of the subject's face changes or part of the face is hidden and the above feature points cannot be recognized, the facial expression conversion stops at an unnatural timing, and only facial images with unnatural conversion can be obtained. In other words, the facial expressions shown in the facial image cannot be converted seamlessly.

This invention has been made in light of the above-mentioned circumstances, and its purpose is to provide an image conversion device, method and program that can seamlessly convert facial expressions shown in facial images.

In order to solve the above problem, an image conversion device according to one embodiment of the present invention comprises a feature point recognition unit that recognizes feature points of facial parts recognized from an image including a human face, a change amount correction unit that corrects a change amount representing the amount of deformation for each of the feature points of the facial parts corresponding to the converted expression when converting the recognized facial expression into a converted expression to be converted, based on the ratio of the angle of the face from the front to the limit angle at which the face in the image cannot be recognized from the front and the proportion of the area excluding areas of the face that are obscured by objects to the entire area of the face, and an expression conversion unit that obtains a converted image in which the human facial expression is converted by deforming the feature points by the corrected change amount.

In order to solve the above problem, an image conversion method according to one embodiment is a method performed by an image conversion device that converts facial expressions in an image of a human face, and includes: recognizing, by a feature point recognition unit of the image conversion device, feature points of facial parts recognized from an image including a human face; correcting, by a change amount correction unit of the image conversion device, a change amount representing the amount of deformation for each of the feature points of the facial parts corresponding to the converted expression when converting the recognized facial expression into a converted expression to be converted, based on a ratio of the angle of the face from the front to the limit angle at which the face in the image cannot be recognized from the front and a ratio of an area excluding areas where the face is hidden by objects to the entire area of the face; and obtaining a converted image in which the facial expression of the person is converted by transforming the feature points by the corrected change amount, by a facial expression conversion unit of the image conversion device.

The present invention makes it possible to seamlessly transform facial expressions appearing in facial images.

FIG. 1 is a block diagram showing an example of the configuration of an image conversion device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of the hardware configuration of the image conversion device. FIG. 3 is a diagram showing an example of facial feature points. FIG. 4 is a diagram showing an example of a storage format of feature points. FIG. 5 is a diagram showing an example of a storage format of the amount of change. FIG. 6 is a flow chart showing an example of an image conversion processing operation by the image conversion device. FIG. 7 is a diagram showing an example of a neural network used by the display ratio calculation unit. FIG. 8 is a diagram showing an example of a grid cell (grid area) processed by the display ratio calculation unit.

[One embodiment]
An embodiment of the present invention will now be described with reference to the drawings.
(Configuration example)
FIG. 1 is a block diagram showing an example of the configuration of an image conversion device according to an embodiment of the present invention.
In the example shown in Figure 1, an image conversion device 100 according to one embodiment of the present invention has an image acquisition unit 11, a feature point recognition unit 12, a face angle calculation unit 13, a display ratio calculation unit 14, a converted facial expression input unit 15, a change amount storage unit 16, a change amount correction unit 17, a facial expression conversion unit 18, and an image output unit 19.

The image acquisition unit 11 acquires a facial image of a user from, for example, an image captured by a web camera or an avatar. The image acquisition unit 11 outputs the acquired facial image to the feature point recognition unit 12, the display ratio calculation unit 14, and the facial expression conversion unit 18.

The feature point recognition unit 12 receives the facial image acquired by the image acquisition unit 11 as input, and recognizes the feature points of the facial parts recognized from the facial image. The method of recognizing feature points in the feature point recognition unit 12 will be described later. The feature point recognition unit 12 outputs the recognized feature points to the face angle calculation unit 13 and the change amount correction unit 17.

The face angle calculation unit 13 receives as input the feature points recognized by the feature point recognition unit 12, calculates the face angle in the face image, for example the angle between the current position of the centre of the face and the position when the face is facing forward (sometimes referred to as the face angle from the front), and outputs data on this calculated angle to the change amount correction unit 17.

The display ratio calculation unit 14 receives the facial image acquired by the image acquisition unit 11 as input, calculates the ratio of the hidden part of the entire face for that facial image, and outputs the calculated ratio data to the change amount correction unit 17.

The conversion facial expression input unit 15 acquires a conversion facial expression (sometimes called a conversion facial expression to be converted), which is the facial expression to be converted to, such as a smile, that is specified and input by the user from a user interface such as a keyboard. The conversion facial expression input unit 15 outputs the acquired conversion facial expression to the change amount correction unit 17.

The change amount storage unit 16 pre-stores (memorizes) a change amount representing the amount of deformation (amount of movement of coordinate values) for each feature point for each facial expression to be converted. The change amount is information indicating how much the coordinate values of each feature point should be moved according to the facial expression to be converted. The change amount can be determined in advance, for example, by a user applying facial expression transformation processing to an expressionless face for a specific facial image, while adjusting it so that the facial expression becomes natural.

The change amount correction unit 17 receives as input the feature points recognized by the feature point recognition unit 12 , the face angle calculated by the face angle calculation unit 13 , and the display ratio calculated by the display ratio calculation unit 14 .
Furthermore, the change amount correction section 17 reads out from the change amount storage section 16 the amount of change corresponding to the facial expression to be converted, which is indicated by the converted facial expression input from the converted facial expression input section 15 .
The change amount correction unit 17 calculates the amount of change by correcting the amount of change in the facial expression to be converted based on the input feature points, face angle, and display ratio using a formula described later, and outputs the calculated amount of change data to the facial expression conversion unit 18.

The facial expression conversion unit 18 receives the amount of change corrected by the change amount correction unit 17 as input. The facial expression conversion unit 18 moves each feature point in the input facial image based on the corrected amount of change, i.e., the amount of change representing the amount of deformation corresponding to the converted facial expression to be converted, based on the input movement amount that is the corrected amount of change for that feature point, thereby obtaining a facial image in which the facial expression of the facial image has been converted. The facial expression conversion unit 18 outputs the converted facial image to the image output unit 19.

The image output unit 19 receives the converted facial image from the facial expression conversion unit 18 as input, and outputs the input facial image. Here, output includes, for example, storing the facial image in a storage medium, displaying the facial image on a display, transmitting the facial image to another device via a communication network, and the like.

FIG. 2 is a diagram showing an example of the hardware configuration of the image conversion device 100. As shown in FIG.
The image conversion device 100 is composed of a computer such as a personal computer, a smartphone, or a server computer. As shown in FIG. 2, the image conversion device 100 has a hardware processor (sometimes simply referred to as a processor) 111A such as a CPU (Central Processing Unit). The CPU can execute multiple information processes simultaneously by using a multi-core and multi-threaded one. The processor 111A may also include multiple CPUs. In the image conversion device 100, a program memory 111B, a data memory 112, a communication interface 114, and an input/output interface 113 are connected to the processor 111A via a bus 115.

The communication interface 114 may include, for example, one or more wired or wireless communication modules. The communication interface 114 may communicate with other computers and web cameras connected via a network (NW) such as a cable, a local area network (LAN), or the internet.

An input device 200 and an output device 300 are connected to the input/output interface 113. The input device 200 includes input devices such as a keyboard, a pointing device such as a mouse, and a sensor device such as a camera. The output device 300 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display. The input device 200 and the output device 300 can also be a so-called tablet type input/display device. This type of input/display device is configured by arranging an input detection sheet using an electrostatic method or a pressure method on the display screen of a display device using, for example, liquid crystal or organic EL (Electro Luminescence). The input/output interface 113 inputs operation information inputted in the input device 200 to the processor 111A, and displays display information generated by the processor 111A on the output device 300.

The input device 200 and the output device 300 do not have to be connected to the input/output interface 113. The input device 200 and the output device 300 can transmit and receive information between the processor 111A and the input device 200 and the output device 300 by being provided with a communication unit for connecting to the communication interface 114 directly or via a network.

In addition, the input/output interface 113 may have a read/write function for a recording medium such as a semiconductor memory such as a flash memory, or may have a connection function with a reader/writer having a read/write function for such a recording medium. Furthermore, the input/output interface 113 may have a connection function with other devices.

The program memory 111B is a non-transient tangible computer-readable storage medium that is a combination of a non-volatile memory that can be written and read at any time and a non-volatile memory that can only be read at any time. Examples of non-volatile memories that can be written and read at any time include HDDs (Hard Disk Drives) and SSDs (Solid State Drives). Examples of non-volatile memories that can only be read at any time include ROMs (Read Only Memory). This program memory 111B stores programs, such as an image conversion program, that are necessary for the processor 111A to execute various control processes according to one embodiment. That is, the processing function units in each of the above-mentioned image acquisition unit 11, feature point recognition unit 12, face angle calculation unit 13, display ratio calculation unit 14, conversion facial expression input unit 15, change amount correction unit 17, facial expression conversion unit 18, and image output unit 19 can all be realized by having the processor 111A read and execute the image conversion program stored in the program memory 111B. It should be noted that some or all of these processing functions may be implemented in a variety of other forms, including integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

Data memory 112 is a tangible computer-readable storage medium that is, for example, a combination of the above-mentioned non-volatile memory and a volatile memory such as a RAM (Random Access Memory). This data memory 112 is used to store various data acquired and created in the course of various processes. In other words, areas are secured in data memory 112 for storing various data as appropriate in the course of various processes.

Figure 3 is a diagram showing an example of facial feature points. The stars in Figure 3 are feature points recognized by processor 111A, and the numbers next to each feature point are unique feature point IDs (IDentifiers) for identifying each feature point. The number of feature point IDs and the part of the face that each feature point ID corresponds to are determined by the feature point recognition method employed. For example, the feature point for feature point ID "18" is predetermined to be the left end of the left eyebrow.

Figure 4 is a diagram showing an example of the storage format of feature points. As shown in Figure 4, the data memory 112 stores the x and y coordinates of feature points in a face image in table format, corresponding to feature point IDs. The coordinate values are in pixels. Therefore, in the example of Figure 3, the data memory 112 stores the x and y coordinates of feature points associated with feature point IDs "1" to "68".

The data memory 112 stores a converted facial expression designated by the user, which is acquired when the processor 111A operates as the converted facial expression input unit 15 described above.
The data memory 112 can store the conversion amount stored in the change amount storage section 16 described above.

Figure 5 is a diagram showing an example of the storage format of the amount of change. As shown in Figure 5, in the data memory 112, the amount of change in the x coordinate and the amount of change in the y coordinate of the feature point are stored in table format for each converted facial expression, associated with the feature point ID, as an amount of change that is independent of the person who is the subject. The value of the amount of change is in pixels. The amount of change is represented by the direction and amount of movement of the feature point. For example, an amount of movement of "+1" represents a movement of one pixel in the positive direction.

The data memory 112 can store a facial image converted when the processor 111A operates as the facial expression conversion unit 18 described above.
Furthermore, the data memory 112 can store various intermediate data generated during the operation of the processor 111A.

(operation)
Next, the operation of the image conversion device 100 will be described.
6 is a flowchart showing an example of the image conversion processing operation by the image conversion device 100. The processor 111A of the image conversion device 100 starts the operation of the image conversion device 100 shown in this flowchart by reading and executing the image conversion program stored in the program memory 111B. Execution of the image conversion program by the processor 111A is started when an instruction to perform image conversion is received from the input device 200 via the input/output interface 113 or via the communication interface 114.

The processor 111A operates as the conversion facial expression input unit 15 and waits for the user to input a designated conversion facial expression, such as a smile, which is the facial expression to be converted to (step S1). For example, the processor 111A determines whether or not the input signal from the input device 200 via the input/output interface 113 or the communication interface 114 includes a designated input of a conversion facial expression. If a designated input of a conversion facial expression is present, the processor 111A proceeds to processing in step S2.

The processor 111A stores the specified converted facial expression in the data memory 112 (step S2).

The processor 111A operates as the image acquisition unit 11 to acquire a facial image (step S3). For example, the processor 111A acquires an image of the subject's face captured by the camera of the input device 200 via the input/output interface 113. Alternatively, the processor 111A acquires a facial image captured by a web camera connected to a network or the face of an avatar generated by another computer via the communication interface 114. The processor 111A stores the acquired facial image in the data memory 112.

The processor 111A operates as the feature point recognition unit 12 and recognizes feature points from the face image stored in the data memory 112 (step S4). The processor 111A recognizes feature points from the face image, for example, by using the face_landmark_detection function of dlib (see, for example, http://dlib.net/face_landmark_detection.py.html). Specifically, the processor 111A extracts a distribution of the gradient direction of brightness, called a HOG (Histogram of Oriented Gradients) feature, from the input face image. A model trained based on data linking the HOG feature with the positions of the facial feature points is generally provided. Therefore, the processor 111A inputs the extracted HOG feature into this learning model and acquires the positions of the facial feature points. The processor 111A stores the acquired positions of the feature points in the data memory 112.

The processor 111A operates as the face angle calculation unit 13 and calculates the face angle in the face image using, for example, opencv (step S5).
Specifically, the processor 111A measures in advance the three-dimensional positions (P_3d) of the feature points of the facial features when the face is facing forward, and stores this in the data memory 112.
The processor 111A obtains the two-dimensional position (P'_2d) of the current feature point of the facial part of the facial image.
The processor 111A calculates the two-dimensional position (P_2d) of the feature point of the facial feature when the above-mentioned three-dimensional position (P_3d) is rotated or moved.
The processor 111A calculates each of the two-dimensional positions by using, for example, the ProjectPoints2 function of opencv (see, for example, http://opencv.jp/opencv-2svn/py/camera_calibration_and_3d_reconstruction.html#projectpoints2).

The processor 111A calculates the sum of squares (D) of the distance between the two-dimensional position (P_2d) and the two-dimensional position (P'_2d).
The processor 111A determines the angle (and amount of movement) that minimizes this sum of squares D by global optimization.

The processor 111A calculates the angle (and amount of movement) that minimizes the above-mentioned angle as the face angle (a) from the front, for example, using the solvPnP function of opencv (see, for example, http://opencv.jp/opencv-2svn/cpp/camera_calibration_and_3d_reconstruction.html#cv-solvepnp).

The processor 111A activates the face recognition tool and moves the face to obtain the position of the feature point at which recognition becomes impossible, thereby calculating in advance the limit face angle (A) at which recognition is possible as an angle that is independent of the person being photographed, and stores this in the data memory 112.

Next, the processor 111A operates as the display ratio calculation unit 14 to calculate the display ratio of the face, which is the ratio of the area of the entire face that is hidden by objects other than the face, for the face image (step S6). For example, if 10% of the entire face is hidden by objects other than the face, the display ratio of the face is 10%.

Here, an example of calculation by the display ratio calculation unit 14 is explained with reference to Figures 7 and 8.

Figure 7 shows an example of a neural network used by the display ratio calculation unit. Figure 8 shows an example of a grid cell processed by the display ratio calculation unit. Here, an example is described for an input image containing animals and various objects, but it can be similarly applied when these are human faces and objects obscuring the faces, such as hands or other objects.

7 and 8, the well-known YOLO (You Only Look Once) (a general object detection method using deep learning) may be used. This method is disclosed in, for example, the following document:
“Joseph Redmon, et al., “YOLOv3: An Incremental Improvement”, arXiv preprint, arXiv:1804.02767, 2018.”

In this technique, processor 111A resizes the face image to a square and inputs it into a CNN (Convolutional Neural Network), a neural network widely used in the field of image processing, as shown in Figure 7. Processor 111A extracts features from the face image through the 24 convolutional layers (Conv. Layer) and 4 pooling layers (see symbol a in Figure 7) in the CNN shown in Figure 7, and can estimate the bounding box of an object in the image and the probability of the object type in the 2 fully connected layers (Conn. Layer) (see symbol b in Figure 7). The final output size of the convolutional layer, 7 x 7, matches the number of divisions in the grid cell.

The input image is divided into S×S grid cells as shown in FIG. 8 (see FIG. 8(a)).
The processor 111A estimates the bounding boxes of B objects for each of the divided grid cells. For each bounding box, the processor 111A outputs a total of five values, including the coordinate values, width, and height (x, y, w, h) of the bounding box, and a confidence score that the bounding box is an object (see FIG. 8B).

The coordinates x and y are the center coordinates of the bounding box based on the boundary of the grid cell, the width w and height h are relative values to the size of the entire image, and the confidence score (score) represents the probability that the bounding box is an object or background. This probability is "1" if it is an object and "0" if it is background.

One index for measuring the accuracy of object region estimation is IoU (Intersection over Union), which indicates the degree of agreement between the correct bounding box and the estimated bounding box. In the above YOLO, the reliability score of the bounding box represents IoU.

The processor 111A estimates the probability of the type of object for each grid cell. For example, in the classification class of type C, when a grid cell is an object, the processor 111A estimates the probability of which class it belongs to, i.e., the conditional probability (see (c) of Figure 8).

The processor 111A combines the class probability estimated here with the bounding box described above to obtain multiple bounding boxes indicating what the object is (see (d) in Figure 8).

The processor 111A selects these bounding boxes, including overlapping regions, using a technique called Non-Maximum Suppression (NMS) based on the bounding boxes with high confidence scores (see (e) in Figure 8). NMS suppresses regions with large IoU values (high degree of overlap) using a threshold value. This results in the detection of object regions.

When there is a face area and an object area overlapping this area, the processor 111A can calculate the display ratio of the face by dividing the area of the overlapping area by the area of the face area.

Next, the processor 111A operates as a change amount correction unit 17, reads out the change amount corresponding to the facial expression to be converted from the change amount storage unit 16, and calculates a change amount that is a correction of the read out change amount corresponding to the facial expression to be converted based on the feature points recognized in S4, the facial angle calculated in S5, and the display ratio calculated in S6 (step S7).

Specifically, processor 111A obtains the face angle, i.e., the face angle a from the front and the limit face angle A at which the face can be recognized, and the proportion H of the area where the face is hidden relative to the entire face area, and based on these, attenuates the amount of change in facial expression transformation, i.e., corrects the amount of change, using the following equation (1), and stores this corrected result in data memory 112.
ΔP _new = ΔP・(1-H)・a/A…Formula (1)
The left side of equation (1), ΔP _new , is the amount of change after attenuation, that is, correction, of facial expression transformation, and ΔP on the right side is the amount of change before correction of facial expression transformation.

That is, in the above example, the amount of change after correction is calculated based on (1) the ratio a/A between the face angle a from the front and the limit face angle A at which the face can be recognized, and (2) the proportion H of the area where the face is hidden to the entire face area.
In addition, without being limited to this example, for example, within the range of allowable accuracy, the amount of change after correction may be calculated based on either (1) the ratio a/A between the face angle a from the front and the limit face angle A at which the face can be recognized, or (2) the proportion H of the area where the face is hidden to the entire face area.

By correcting the amount of change in this way, even if feature points cannot be recognized due to a change in the angle of the face or part of the face being hidden, facial expression conversion will not stop at an unnatural time, and the facial expression in the facial image can be converted naturally.

The processor 111A operates as the facial expression conversion unit 18 and converts the facial expression of the facial image stored in the data memory 112 (step S8). That is, the processor 111A converts the facial image based on the result of correcting the amount of change according to the converted facial expression stored in the data memory 112. For example, the processor 111A uses an implementation of MLS (see, for example, https://github.com/Jarvis73/Moving-Least-Squares) or the like.

Specifically, processor 111A moves each feature point by the amount of change after correction of the amount of change corresponding to the converted facial expression stored in data memory 112. For example, when converting a facial expression to a smile, the x and y coordinates before conversion for the control point with feature point ID "1" are (23, 45) (see Figure 4), so processor 111A performs a conversion such that the pixel of the feature point moves to (24, 47) by incrementing the x coordinate by "+1" and the y coordinate by "+2" (see Figure 5).

Then, for the feature points, the processor 111A applies an affine transformation (including the Helmert transformation = similarity transformation and rigid deformation) shown in the following equation (2).

In the above formula (2), x and y are the coordinates of the nearby feature point, x' and y' are the coordinates obtained by adding the amount of change to the coordinates of the feature point, a, b, c, and d are parameters, and t _x and t _y are translation parameters. The processor 111A calculates the least square means of the coordinates x' and y' obtained by adding the amount of change to the coordinates of the feature point x and y, and obtains parameters a, b, c, d, t _x , and t _y that minimize this by global optimization. Then, the processor 111A determines the coordinates after transformation using these obtained parameters, with the coordinates of the target point to be transformed being x and y. The processor 111A determines the coordinates after transformation from the feature point by the above affine transformation using the parameters a, b, c, d, t _x , and t _y thus obtained.

The processor 111A stores the facial image thus converted as a converted image in the data memory 112.

The processor 111A operates as the image output unit 19 and outputs the converted image stored in the data memory 112 (step S9). For example, the processor 111A causes the output device 300 to display the facial image via the input/output interface 113. Alternatively, the processor 111A transmits the facial image onto the network via the communication interface 114 and causes it to be displayed on a display device connected to the network, or on the display unit of another computer connected to the network.

The processor 111A determines whether or not to terminate the operation of the image conversion device 100 shown in the flowchart of FIG. 6 (step S10). For example, the processor 111A checks whether or not the user has instructed the end of image conversion from the input device 200, via the input/output interface 113, or via the communication interface 114. If the operation is to be terminated (YES in step S10), the processor 111A terminates the operation shown in the flowchart of FIG. 6.

On the other hand, if the above operation has not yet ended (NO in step S10), processor 111A operates as conversion facial expression input unit 15 and determines whether or not the user has input a change to the conversion facial expression (step S11). If there has been no input specifying a change to the conversion facial expression (NO in step S11), processor 111A proceeds to processing in step S3. If there has been input specifying a change to the conversion facial expression (YES in step S10), processor 111A proceeds to processing in step S2.

The image conversion device 100 according to the embodiment described above includes a face angle calculation unit 13, a display ratio calculation unit 14, a change amount correction unit 17, and an expression conversion unit 18. The expression conversion unit 18 obtains a converted image in which a human facial expression is converted by converting feature points using a deformation amount corresponding to a conversion expression to be converted.
Therefore, the image conversion device 100 according to one embodiment can convert the facial expression of a facial image naturally without stopping facial expression conversion at an unnatural timing, even if feature points cannot be recognized due to a change in the angle of the face or part of the face being hidden.

[Other embodiments]
It should be noted that the present invention is not limited to the above embodiment.
For example, the flow of each process described above is not limited to the procedures described, and the order of some steps may be changed, or some steps may be performed simultaneously in parallel.

In addition, the process flow described above is for converting facial expressions in facial images acquired in real time in real time, but it can also be applied to applications where the facial expressions of stored facial images are converted rather than real-time processing.

The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), semiconductor memory (ROM, RAM, flash memory, etc.), or may be distributed by transmission via a communication medium. The programs stored on the medium include a setting program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program recorded on the recording medium, and in some cases, constructs the software means using the setting program, and executes the above-mentioned processing by controlling the operation of the software means. The recording medium referred to in this specification is not limited to a recording medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory provided inside the computer or in a device connected via a network.

Note that the present invention is not limited to the above-described embodiments, and can be modified in various ways in the implementation stage without departing from the gist of the invention. The embodiments may also be implemented in appropriate combination, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combinations selected from the multiple constituent elements disclosed. For example, if the problem can be solved and an effect can be obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiments, the configuration from which these constituent elements are deleted can be extracted as an invention.

100: Image conversion device 11: Image acquisition section 12: Feature point recognition section 13: Face angle calculation section 14: Display ratio calculation section 15: Converted facial expression input section 16: Change amount storage section 17: Change amount correction section 18: Facial expression conversion section 19: Image output section 111A: Processor 111B: Program memory 112: Data memory 113: Input/output interface 114: Communication interface 115: Bus 200: Input device 300: Output device

Claims (6)

a feature point recognition unit that recognizes feature points of facial parts from an image including a human face;
a change amount correction unit which corrects a change amount representing an amount of deformation for each of the feature points of the facial features according to the converted facial expression when converting the recognized facial expression into a converted facial expression to be converted, based on a ratio of an angle of the face from the front to a limit angle at which the face in the image cannot be recognized from the front, and a ratio of an area excluding areas of the face hidden by objects to an entire area of the face;
an expression conversion unit that obtains a converted image in which the facial expression of the person is converted by transforming the feature points according to the corrected change amount;
An image conversion device comprising: The change amount correction unit
correcting the amount of change by multiplying a predetermined amount of change for each of the feature points of the facial features by a ratio of an angle of the face from the front to a limit angle at which the face in the image cannot be recognized from the front, and a ratio of an area of the face excluding an area of the face hidden by an object to a total area of the face;
2. The image conversion device according to claim 1. calculating a two-dimensional position of a feature point of the facial part when the three-dimensional position of the feature point of the facial part when the face is facing forward is rotated or moved, and calculating an angle at which a sum of squares of a distance between the calculated two-dimensional position and a current two-dimensional position of the feature point of the facial part is a minimum as an angle of the face from the front;
2. The image conversion device according to claim 1. a storage device in which a change amount representing a deformation amount for each of the feature points is stored in advance for each of the facial expressions to be converted;
a conversion expression input unit for inputting the conversion expression to be converted;
Further comprising:
The change amount correction unit
reading out the amount of change corresponding to the input converted facial expression from the storage device, and correcting the amount of change that has been read out;
4. An image conversion device according to claim 1. 1. A method performed by an image transformation device for transforming facial expressions in an image of a human face, comprising:
Recognizing feature points of facial features from an image including a human face by a feature point recognition unit of the image conversion device;
a change amount correction unit of the image conversion device corrects a change amount representing a deformation amount for each of the feature points of the facial features according to the converted facial expression when converting the recognized facial expression into a converted facial expression to be converted, based on a ratio of an angle of the face from the front to a limit angle at which the face in the image cannot be recognized from the front, and a ratio of an area excluding areas of the face hidden by objects to an entire area of the face;
obtaining a converted image in which the facial expression of the person is converted by transforming the feature points by the corrected change amount using a facial expression conversion unit of the image conversion device;
13. An image conversion method comprising: 5. An image conversion processing program that causes a processor to function as each unit of the image conversion device according to claim 1.

Images