JP2886933B2 - Facial expression measuring device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a facial expression measuring device that measures facial expressions by a computer based on a time-series image of facial expressions and performs machine recognition of facial expressions.

[Conventional technology]

In the field of psychology, for example, as described in “Kudo Riki Translation Edition“ Introduction to Expression Analysis ”, Hikarisho Seishin Shobo”,
Several methods for analyzing human facial expressions by human inspection have been proposed. In recent years, when sending human facial expressions and actions in systems such as videophones and videoconferencing, those that communicate as symbol data such as "laughing" instead of image signal form are being used. Showa
In the D-97 of IEICE Spring National Convention, “Expression Analysis of Facial Images in Analysis-Synthesized Image Coding” by D-97, one method of facial expression analysis was proposed, which was characterized by facial features. Is an attempt to recognize facial expressions from the changes.

[Problems to be solved by the invention]

In the case of this proposal, there are drawbacks that it is difficult to automatically detect these feature points with high accuracy, and that the locations that can be feature points are limited, so that it is not always possible to capture even delicate movements of muscles. In addition, there is a problem that these data have low relevance to the movements of the muscles controlling the facial expressions, and have not been analyzed physiologically.

An object of the present invention is to provide a highly accurate facial expression measuring device that measures delicate movements of muscles over the entire face.

[Means for solving the problem]

The most important feature of the present invention is that a circuit for extracting a motion vector called an optical flow is provided as a component. This circuit is different in that it can detect the movement of muscle even in a place where there is no clear line segment or end point, such as a cheek, which cannot be performed by the conventional technology.

〔Example〕

Hereinafter, the operation of the device of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram for explaining the overall configuration of an expression measuring apparatus according to the present invention. Reference numeral 101 denotes a video input device including a television camera, an A / D conversion circuit, and the like. In the expression measuring apparatus of the present invention, the noise removing circuit 102, the muscle position window designating module 10
3, motion vector detection circuit 104, motion vector averaging circuit 10
5, measurement result output terminal 106, time-series image data input terminal 107
Having.

In general, digital image data captured by the video input device 101 or the like is input to the apparatus. To operate this, the digital image output of the video input device 101 is connected to the noise removal circuit 102.
To remove noise by local averaging in the temporal and spatial directions of the image or by a Gaussian filter. Next, the image sequence is input to a motion vector detection circuit 104 to detect a motion vector. Here, the detection of the motion vector is performed only for the area specified by the muscle position window specifying module 103, and each point is processed. Is calculated. The motion vector averaging circuit 105 weights and averages the motion vector of each point in the specified area to obtain a motion vector. After calculating the muscle direction vector data from the muscle position window designation module 103 Get the inner product Is calculated and output to the measurement result output terminal 106. This output is the measured amount of movement of the specified muscle.

In the above description, the case of detecting the movement of one muscle has been described. However, the device may be configured to detect a plurality of muscles by specifying a plurality of muscles with the muscle position window specifying module 103. Good.

Next, the muscle position window designation module 103 will be described. First, the muscle position window designation module referred to in claim (1) has a configuration as shown in FIG. 201 is a control circuit,
202 is a display, 203 is a pointing device such as a mouse, 204 is a muscle position window displayed on the display, and 205 is a target face image also displayed on the display. 206 is an image input terminal, 207 is a window area coordinate output terminal, and 208 is a muscle direction vector output terminal. Here, the operator of this device uses the mouse 203 while watching the image of the face image 205 displayed on the display 202,
The muscle position window 204 is indicated. At the same time, specify the direction of the muscle. The control circuit 201 is a circuit that controls the display 202 and the mouse 203 to provide the above-mentioned functions, and also outputs the designated window coordinates to the window area coordinate output terminal 207 and the muscle direction vector to the muscle direction vector output terminal 208. Output. In some cases, the operator repeatedly specifies the position of the window and sends information about a plurality of windows from the output terminals 207 and 208. In this way, the muscle position window designation module 103 provides a function that allows the operator to designate a window interactively, and enables measurement of movement of a specific muscle.

Next, FIG. 3 shows a configuration of a muscle position window designation module according to claim (2). 301 is a control circuit, 302 is a face image processing device, 303 is an image input terminal, 304 is a window area coordinate output terminal, and 305 is a muscle direction vector output terminal.

Here, the face image processing device 302 is a device that automatically processes the function of interactively specifying a muscle position window described with reference to FIG. Therefore, the control circuit 301 receives the image data from the image input terminal 303 and transfers it to the face image processing device 302, receives the coordinates of the muscle position window and the direction vector as the processing result, and outputs the window area coordinate output terminal 304 and the muscle direction vector, respectively. Output to the output terminal 305. In this manner, the muscle position window specifying module automatically sets the muscle position window by the face image processing apparatus, so that even when a different object appears, the movement of a specific muscle can be automatically measured.

FIG. 4 is a diagram for explaining the operation of the face image processing apparatus.
Reference numeral 401 denotes a face part extraction circuit, 402 denotes a coordinate normalization circuit, 403 denotes a muscle position direction table, 404 denotes an image data input terminal, and 405 denotes a window data output terminal. Here, the face image data enters the face part extraction circuit 401 from the image data input terminal 404, and the face part extraction circuit 401 extracts parts such as eyes, nose and mouth by image processing. The coordinate normalization circuit normalizes these component positions.
402. In the muscle position direction table 403, window coordinates and direction vectors corresponding to each muscle are stored in a table format in a form normalized to a general face shape in advance. The coordinate normalization circuit 402 looks at the parameters used for image normalization and the contents of the muscle position direction table 403, and inversely converts the window corresponding to each muscle position into the coordinate system of the original image to extract the face part.
Output from the window data output terminal 405 through 401. As described above, the window can be automatically set from the image by the coordinate normalization circuit 402 and the muscle position direction table 403 in which the data of the position corresponding to each muscle is tabulated. The extraction direction of facial parts is described in, for example, IEICE Transactions, April 1973, v
ol.56-D, no.4, Sakai et al.
Can be performed by using the method disclosed in US Pat.

[Effects of the Invention] As described above, according to the present invention, the movement of the facial expression muscle can be measured in a non-contact manner, so that it can be used as an expression analysis module in image communication, and in the fields of psychology and medicine. In this case, there is an advantage that subtle muscle movement can be automatically measured without contact.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining the configuration of the entire facial expression measuring apparatus of the present invention, FIG. 2 is a configuration of a muscle position window designation module used in claim (1), and FIG. 3 is a component in claim (2). FIG. 4 is a diagram for explaining the operation of the face image processing apparatus, showing the configuration of the muscle position window designation module. In the figure, 101 is a video input device, 102 is a noise removal circuit, 103 is a muscle position window designation module, 104 is a vector detection circuit, 105 is a motion vector averaging circuit, 201 is a control circuit, 202 is a display, and 203 is pointing Device, 204 is a muscle position window, 205 is a face image, 301 is a control circuit, 302 is a face image processing device, 4
01 is a face part extraction circuit, 402 is a coordinate normalization circuit, and 403 is a muscle position direction table.

Claims (2)

(57) [Claims] A facial expression measuring device for processing a time-series image obtained by photographing a face presenting a facial expression with a video input device, comprising: a noise removal circuit; A circuit, a muscle position window specifying module, and a motion vector averaging circuit. The time series image is input to the noise removing circuit to remove noise, and the result is input to the motion vector detecting circuit. Calculates the motion vector of each point on the image sequence for the window region specified by, and outputs the result of averaging the specific direction components in each window by the motion vector averaging circuit from the result to create a facial expression An expression measuring device characterized by measuring muscle movement. 2. The facial expression measuring device according to claim 1, wherein the muscle position window specifying module detects each component of the face by image processing of the face image, and then detects a relative position with the component specified in advance. A facial expression measuring device that automatically measures the movement of the muscles that create facial expressions based on the physical positional relationship.