JP6952298B2 - Line-of-sight conversion device and line-of-sight conversion method - Google Patents

Description

The present disclosure relates to a line-of-sight conversion technique for changing the direction of the line of sight of a photographed person in an image.

In a video dialogue system, the position of the camera and the position of the screen of the display are usually different. Therefore, the user cannot match the line of sight with the conversation partner who is looking at the screen, and similarly, the conversation partner cannot match the line of sight with the user. For a more natural dialogue, a video dialogue system that matches the eyes is desired.

Patent Document 1 describes that a half mirror is added to acquire a front-facing face image. Patent Document 2 describes that a front-facing face image is generated by installing one camera on each side of the monitor screen. Patent Document 3 describes that the eye image is corrected so that the position of the pupil is at the center of the eye image.

JP-A-11-177949 Japanese Unexamined Patent Publication No. 8-251562 Japanese Unexamined Patent Publication No. 2015-149016

However, if hardware such as a half mirror or a camera is added, there is a problem that the system becomes large. Further, in the case of correcting an image as in Patent Document 3, if the processing is simply performed for each frame, the position of the eyes may differ for each frame, and an unnatural moving image may be obtained.

An object of the present invention is to make it possible to obtain a more natural moving image without increasing the size of the system when seeking an image in which the line of sight of the photographed person is changed, particularly a moving image.

The line-of-sight conversion device according to the present disclosure stores a camera that acquires an image of the photographed person's face as a processing target image, and a plurality of images of the photographed person's face whose line of sight is directed toward the photographing device as reference images. A reference image storage device, a shape model generator that generates a shape model for the shape of the photographed person's eyes based on the plurality of the reference images, and feature points of the photographed person's eyes in the processed image. It is defined by a feature point extractor that extracts the image, a feature point position corrector that corrects the position of the feature point using the shape model, and the feature point whose position is corrected by the feature point position corrector. It has an image corrector that transfers the corresponding region of the reference image to the region and corrects the image to be processed so that the line of sight of the photographed person appears to be directed toward the camera.

In the line-of-sight conversion method according to the present disclosure, an image of the photographed person's face is acquired as a processing target image, and a plurality of images of the photographed person's face whose line of sight is directed toward the photographing device are stored as reference images, and a plurality of images are stored. Based on the reference image of the above, a shape model for the shape of the eye of the photographed person is generated, feature points of the eye of the photographed person in the image to be processed are extracted, and the shape model is used to describe the above. The position of the feature point is corrected, the corresponding region of the reference image is transferred to the region defined by the feature point whose position has been corrected, and the line of sight of the photographed person is directed toward the camera. The image to be processed is corrected so as to appear as.

According to these line-of-sight conversion devices and line-of-sight conversion methods, it is possible to obtain an image in which the line of sight appears to be directed to the camera even if the subject is not facing the direction of the camera. Therefore, to the dialogue partner, it seems that the line of sight is directed to oneself, and a natural dialogue becomes possible. Since the shape model is used, the position of the eyes is stable especially in the moving image, and a more natural moving image can be obtained. Since it is not necessary to add a half mirror or the like, the system does not become large. Further, since the processing for the two-dimensional image is performed, the calculation cost can be kept relatively small.

According to the present disclosure, it is possible to obtain a more natural moving image without increasing the size of the system.

FIG. 1 is a block diagram showing a configuration example of a line-of-sight conversion device according to an embodiment of the present invention. FIG. 2 is a flowchart showing an example of a shape model generation process in the line-of-sight conversion method according to the embodiment of the present invention. FIG. 3 is an explanatory diagram showing an example of the feature points of the eyes. FIG. 4 is a diagram showing an example of eye shape change when the coefficient corresponding to each eigenvector is changed in the eigenspace constructed by using the eigenvectors. FIG. 5 is a diagram showing an example of the relationship between the number of eigenvectors used (the number of bases) and the cumulative contribution rate. FIG. 6 is a flowchart showing an example of processing for a target image in the line-of-sight conversion method according to the embodiment of the present invention. FIG. 7 is a diagram showing an example of extracted feature points and eye regions in the image to be processed. FIG. 8 is a diagram showing an example of extracted feature points and eye regions in the reference image. FIG. 9 is a flowchart showing in more detail an example of the process of correcting the position of the feature point of FIG. FIG. 10 is a diagram showing an example in the case where the positions of the extracted feature points in the image to be processed are not correct. FIG. 11 is a diagram showing an example of a feature point whose position has been corrected. FIG. 12 is an explanatory diagram showing an example of the shape and feature points of the eyes in the reference image. FIG. 13 is a diagram showing an example of feature points rearranged around the eyes. FIG. 14 is an example of the image to be processed before processing. FIG. 15 is an example of an image obtained by performing the processing of FIG. 6 on the image of FIG. FIG. 16 is a graph showing an example of the transition of the correlation value. FIG. 17 is a block diagram showing a configuration example of a computer system that realizes the line-of-sight conversion device according to the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a configuration example of a line-of-sight conversion device according to an embodiment of the present invention. The line-of-sight conversion device 10 of FIG. 1 includes a reference image storage device 12, a shape model generator 14, a camera 16, a feature point extractor 18, a feature point position corrector 20, an image corrector 22, and an image output. It has a vessel 28 and. The image corrector 22 has a texture synthesizer 24 and a shape changer 26.

The reference image storage device 12 stores a plurality of images of the face of the photographed person whose line of sight is directed toward the photographing device as a reference image. These images are captured and stored in advance. The shape model generator 14 generates a statistical shape model (hereinafter, simply referred to as a shape model) for the shape of the eyes of the photographed person based on a plurality of reference images. The shape of the eye is represented by the feature points of the eye.

The camera 16 acquires an image of the face of the person to be photographed as an image to be processed. The feature point extractor 18 extracts the feature points of the eyes of the photographed person in the image to be processed. The feature point position corrector 20 corrects the position of the feature point in the image to be processed by using the shape model generated by the shape model generator 14. The image corrector 22 transfers the corresponding area of the reference image to the area defined by the feature points whose positions have been corrected by the feature point position corrector 20, so that the line of sight of the photographed person faces the direction of the camera 16. Correct the image to be processed so that it looks like it is.

The user who is the photographed person talks with the conversation partner while looking at the screen of the television receiver or the display connected to the computer. The user's television receiver or computer is connected to the television receiver or computer of the dialogue partner via a communication network such as the Internet. The user's face is photographed by the camera 16, and the image is processed by the line-of-sight conversion device 10 and then transmitted to the dialogue partner. In the following, the camera 16 will be described as being arranged on the display, for example, but the camera 16 may be arranged elsewhere near the display or built into the television receiver. Such a case can be described in the same manner.

FIG. 2 is a flowchart showing an example of a shape model generation process in the line-of-sight conversion method according to the embodiment of the present invention. First, the generation of the shape model will be described. In the block 112, for example, the camera 16 takes a plurality of images of the user's face, and the reference image storage device 12 stores these images. At this time, the user is in almost the same position (for example, sitting) as when talking with the conversation partner while looking at the display. Also, the user's face is facing the display, but the user's line of sight is facing the camera that shoots. Instead of the camera 16, an image of the user's face may be captured by another imaging device. In this case, the positional relationship between the photographing device and the user is the same as in the above case. The number of images stored in the reference image storage device 12 is, for example, 100, but may be more or less.

FIG. 3 is an explanatory diagram showing an example of the feature points of the eyes. In the block 114 of FIG. 3, the feature points of the eyes are extracted from the reference image stored in the reference image storage device 12, the extracted feature points are added to the reference image, and the feature points are stored in the reference image storage device 12. This process is performed for each of the reference images stored in the reference image storage device 12. The feature points are obtained as shown by the white circles in FIG. 3 so as to represent the shape and region of the eyes. In the example of FIG. 3, for each of the left and right eyes, a point substantially at the center of the pupil and eight points on the boundary line of the eye region are obtained as feature points. As points on the boundary line, points at the left and right ends of the eye region, and in addition, three points on the upper boundary line and three points on the lower boundary line are required. In order to find the exact position of the feature point, we will find the feature point manually here, but if the feature point can be extracted with sufficient accuracy at a sufficient calculation cost, it will be automatically obtained. You may try to find the feature points.

In block 116, the shape model generator 14 performs a process of aligning all the reference images. Here, the shape model generator 14 performs translation, rotation, enlargement / reduction of the image so that the position, inclination, and size of the eyes in each reference image are substantially the same. For example, the position of the feature point can be used for alignment.

In block 118, the shape model generator 14 generates a shape model for the shape of the eye by principal component analysis based on the feature points of the user's eyes extracted from each of the plurality of reference images. An example of principal component analysis will be described. For each reference image, consider a u-dimensional vector c = {c ₁ , c ₂ , ..., _cu } for each of the left and right eyes. In the present embodiment, the vector c is an 18-dimensional vector having the coordinates (x-coordinate and y-coordinate) of the nine feature points of the eyes arranged in order as elements (u = 18), and represents the region of the eyes. ing.

The shape model generator 14 obtains an average vector from N vectors c by the following equation 1.

The shape model generator 14 obtains the covariance matrix S from each vector c and the average vector by Equation 2. The covariance matrix S is a u × u matrix.

Geometric model generator 14, with respect to the covariance matrix S, determining the eigenvalues lambda _i and eigenvectors (principal components) v _i by solving the eigenvalue problem (Equation 3).

The shape model generator 14 selects m eigenvectors corresponding to the upper m eigenvalues (8 in this case). The shape model generator 14 stores the selected eigenvectors in, for example, the reference image storage device 12.

Eigenspace _V = configured using the selected eigenvectors _{(v 1, v 2, ...} , v m) in the coefficient vector c vector _{b = (b 1, b 2} , ..., b m) of the formula It can be calculated as in 4. The coefficient vector is a vector having a coefficient for each eigenvector as an element, and is also called a feature vector.

The shape model generator 14 obtains the coefficient vector b using Equation 4. Vector c, using the coefficient vector b, it can be expressed as a linear sum of eigenvector v _i as Equation 5.

Eigenvectors v _i is also known as eye-specific shape, representing the eye shape features. When the number m of the eigenvectors used is smaller than the number u of the dimensions of the vector c, it can be said that the number of dimensions of the original data is reduced by Equation 5. Eigenvectors v _i are respectively, "eye opening degree" and the eyes of shape features such as "eyes that degree" represents a difference from the mean shape, easily eye by increasing or decreasing the coefficient of the respective The shape can be changed. In addition, an unknown eye shape is projected onto this eigenspace using Equation 4, and the characteristics of the individual eye shape (eye opening degree, fishing eye degree, etc.) are determined from the obtained coefficient values. It is also possible to do. Thus, m eigenvectors v _i can be said to be represented a variety of eye shape, a statistical shape model for the eye shape.

Table 1 shows an example of the obtained principal components (eigenvectors v _{1 to v 8).} Table 2 shows an example of the coefficient vector b corresponding to the vector c in Table 1.

Figure 4 is the eigenspace V constructed using eigenvectors a diagram showing an example of eye shape change when changing the coefficients b _i for each eigenvector. The first to third principal components are eigenvectors from the one with the larger amount of information (larger eigenvalue) to the third place among the eigenvectors acquired by the principal component analysis. From FIG. 4, it can be seen that each eigenvector represents a different shape feature of the eye. The change in shape by changing the coefficients b _i for each eigenvector is seen, the position, size, or the inclination it can be seen that there is no change. In FIG. 4, sigma represents a standard deviation of the coefficients b _i of each eigenvector obtained from the reference image used in the principal component analysis, the column of mean ± 0σ represents the mean vector.

FIG. 5 is a diagram showing an example of the relationship between the number of eigenvectors used (the number of bases) and the cumulative contribution rate. In FIG. 5, the cumulative contribution rate R for the right eye and the cumulative contribution rate L for the left eye are shown. As shown in FIG. 5, as the number of eigenvectors used as the shape model increases, the cumulative contribution rate of information in the principal component analysis increases, but the amount of calculation using the shape model thereafter also increases. In this embodiment, 8 is adopted as the number m of the eigenvectors so that the cumulative contribution rate exceeds 95%.

FIG. 6 is a flowchart showing an example of processing for a target image in the line-of-sight conversion method according to the embodiment of the present invention. After the process of FIG. 2 is completed, the process of FIG. 6 is performed. If the process of FIG. 2 is performed once, it is not necessary to perform the process of FIG. 2 thereafter.

In the block 140A of FIG. 6, the image to be processed (first frame) is input. Specifically, the camera 16 captures an image of the user's face and outputs it to the feature point extractor 18.

In block 142, the feature point extractor 18 receives the image to be processed from the camera 16 and reads an appropriate reference image from the reference image storage device 12. Here, the feature point extractor 18 reads out a face image having a direction close to the face orientation in the processing target image, for example, a reference image including a face image having a direction closest to the face orientation in the processing target image. The feature point extractor 18 extracts the feature points of the eyes from the image to be processed and the reference image, for example, as shown in FIG. 3, and outputs the coordinates of these images and the feature points. Such face orientation detection and feature point extraction are possible by methods well known to those skilled in the art. FIG. 7 is a diagram showing an example of extracted feature points and eye regions in the image to be processed. FIG. 8 is a diagram showing an example of extracted feature points and eye regions in the reference image. The eye area is defined as an area surrounded by the extracted feature points.

In block 144A, the feature point position corrector 20 corrects the feature point position in the image to be processed so that the user's eyes have a shape represented by using the shape model. Here, the extracted feature points are used as the initial coordinates, and the shape transformation by the statistical shape model of the eyes is combined with translation, enlargement / reduction, or rotation to make the eyes more appropriate, for example, optimal in the vicinity of the initial coordinates. Search for the feature point coordinates of. Such a search process for correcting the feature point position will be described.

FIG. 9 is a flowchart showing in more detail an example of the process (block 144A) for correcting the position of the feature point of FIG. In the block 160, the feature point position corrector 20 warps the eye region of the image to be processed by the Piecewise-Affine transformation, and makes the number of pixels the same as the number of pixels in the eye region of the reference image. This is because it is necessary to obtain the correlation value between the eye regions of both images in the following processing. Warping may be performed so that the number of pixels in the eye area of the reference image is the same as the number of pixels in the eye area of the image to be processed.

In block 161, the feature point position corrector 20 obtains the correlation value r between the processing target image and the reference image by using Equation 6 as an evaluation function while translating the processing target image in the x-axis direction. ..

ここで、例えば、値ｔ^１，ｔ^２はそれぞれ処理対象画像及び参照画像の輝度値を表し、Ｍは目領域の総画素数を表す。値ｔ^１，ｔ^２は、輝度以外の値、例えば色相や彩度等であってもよい。平行移動の際には、特徴点位置補正器２０は、例えば、目の横幅×0.02のステップで、負の方向に３ステップ、正の方向に３ステップ、処理対象画像を移動させ、移動させる毎に相関値ｒを求める。特徴点位置補正器２０は、相関値ｒが最大となる処理対象画像の位置を求め、その位置に処理対象画像を置く。

Here, for example, the values t ¹ and t ² represent the brightness values of the image to be processed and the reference image, respectively, and M represents the total number of pixels in the eye region. The values t ¹ and t ² may be values other than luminance, such as hue and saturation. In the case of translation, the feature point position corrector 20 moves and moves the image to be processed by, for example, 3 steps in the negative direction and 3 steps in the positive direction in steps of the width of the eyes × 0.02. The correlation value r is obtained from. The feature point position corrector 20 obtains the position of the processing target image having the maximum correlation value r, and places the processing target image at that position.

In block 162, the feature point position corrector 20 obtains the correlation value r between the processing target image and the reference image by using Equation 6 as an evaluation function while translating the processing target image in the y-axis direction. .. In the case of translation, the feature point position corrector 20 moves and moves the image to be processed by, for example, 3 steps in the negative direction and 3 steps in the positive direction in steps of the vertical width of the eyes × 0.02. The correlation value r is obtained for each. The feature point position corrector 20 obtains the position of the processing target image having the maximum correlation value r, and places the processing target image at that position.

In block 164, the feature point position corrector 20 obtains a correlation value r between the processing target image and the reference image by using the equation 6 as an evaluation function while enlarging or reducing the processing target image. At the time of enlargement or reduction, the feature point position corrector 20 enlarges or reduces the image to be processed by, for example, 3 steps in the negative direction and 3 steps in the positive direction in steps of 2% magnification, and enlarges or reduces the image to be processed. The correlation value r is obtained each time the image is reduced. The feature point position corrector 20 obtains the magnification of the processing target image having the maximum correlation value r, and enlarges or reduces the processing target image so as to have the magnification.

In block 166, the feature point position corrector 20 obtains the correlation value r between the processing target image and the reference image by using the equation 6 as an evaluation function while rotating the processing target image. At the time of rotation, the feature point position corrector 20 rotates the image to be processed in 3 steps in the negative direction and 3 steps in the positive direction in a step of inclination of 1 °, and the correlation value is rotated each time the image to be processed is rotated. Find r. The feature point position corrector 20 obtains the inclination of the processing target image having the maximum correlation value r, and rotates the processing target image so as to have the inclination.

At block 168, the feature point position corrector 20 determines the coefficients for the shape model (ie, the coefficient vector b). The feature point position corrector 20 uses Equation 6 as an evaluation function while changing each coefficient of the shape model (that is, changing the coefficient vector b) to correlate between the image to be processed and the reference image. Find the value r. More specifically, the feature point position correcting unit 20, the coefficients b ₁ to first corresponding to the first principal component, for example, in steps of the standard deviation sigma × 0.5, 6 steps in the negative direction, the positive direction 6 steps are changed, and the correlation value r is obtained each time the change is made. Feature point position correcting unit 20 obtains the coefficient correlation value r is maximized, determining the coefficients b ₁ to the value. The standard deviation σ of the coefficient b ₁ is obtained from a plurality of reference images when obtaining the shape model.

After that, the feature point position corrector 20 performs the same processing on the second, third, ..., And eighth principal components in this order to determine the _{coefficients b 2} , b ₃ , ..., B _8. As the standard deviation σ, the standard deviation of the coefficients for each principal component is used. Then, the vector c corresponding to the shape of the eye can be obtained by the above-mentioned equation 5. The feature point position corrector 20 outputs the _{obtained coefficients b 1} , b ₂ , ..., B _8.

Table 3 shows an example of the search range as described above.

FIG. 10 is a diagram showing an example in the case where the positions of the extracted feature points in the image to be processed are not correct. FIG. 11 is a diagram showing an example of a feature point whose position has been corrected. The positions of the feature points extracted by the feature point extractor 18 in the block 142 may not be correct, as shown in FIG. In such a case, if the feature point position is corrected by the process of FIG. 9, the feature point position can be corrected to the correct position as shown in FIG. 11, for example.

The order of processing of blocks 161, 162, 164 and 166 in FIG. 9 may be changed.

Next, in block 146A of FIG. 6, the texture synthesizer 24 transfers the corresponding region of the reference image to the region defined by the corrected feature points, and the user's line of sight is directed toward the camera 16. Correct the image to be processed so that it looks like it is. Specifically, the texture synthesizer 24 performs the following processing. That is, the texture synthesizer 24 uses the feature points to divide the eye region of the image to be processed and the reference image into a triangular region as shown in FIG. The texture synthesizer 24 transfers the texture of the eye region of the reference image to the corresponding region of the processed image by using the Piecewise-Affine transformation for each triangular region. At this time, the affine transformation is performed in each triangular region.

Next, the shape changer 26 corrects the shape of the image processed by the texture synthesizer 24. Since the eye shape of the image to be processed is different from the eye shape of the reference image in which the user's line of sight is directed toward the camera, the result of the texture transfer described above is an unnatural image due to an imbalance between the shape and the texture. It tends to be. Therefore, the shape changer 26 further corrects the shape of the eyes defined by the feature points in the image to be processed to the shape of the eyes in the reference image by warping to realize a natural transfer result.

FIG. 12 is an explanatory diagram showing an example of the shape and feature points of the eyes in the reference image. For example, when the shape of the eyes of the image to be processed is the shape shown in FIG. 3, the shape changer 26 corrects the shape of the eyes to the shape shown in FIG. Regarding this correction, once the relative positions of the camera, display, and user are determined, the required correction (relationship of eye shape between the image to be processed and the reference image) is almost determined. When the camera is installed above the display, this correction mainly corresponds to the process of widening the eye opening. As a warping method, for example, FFD (Free-Form Deformation) is used.

Here, the process for transferring the texture will be further described. FIG. 13 is a diagram showing an example of feature points rearranged around the eyes. The boundary between the transferred texture and the surrounding texture may be unnatural. In the automatic extraction of feature points, some positional errors often occur, and since the textures of different images are transferred in the first place, even if the same person is photographed in the same environment, the processing target image and the reference image will not be the same. This is because there is a certain difference in the brightness value of the corresponding portion. Therefore, in order to reduce the influence of the feature point extraction error, the texture synthesizer 24 increases the distance from the center of the pupil of the feature point coordinates on the contour of the eye after warping by a constant magnification. Then, as shown in FIG. 13, the feature points are rearranged around the eyes to expand the area of the eyes. The area surrounded by the rearranged feature points indicates the enlarged area.

Therefore, when transferring the texture, the texture synthesizer 24 applies a gradation according to the distance from the boundary of the enlarged region. Specifically, the texture synthesizer 24 is included in the reference image so that the image included in the processed image gradually changes to the image included in the reference image as it approaches the center of the eye in the enlarged region. The image is superimposed on the image included in the image to be processed. That is, the weight of the image to be processed is increased near the boundary, and the weight of the reference image is gradually increased as it is closer to the center of the eye. Since we want to retain the texture of the reference image inside the eye, the gradation is almost complete outside the eye, i.e., in the area magnified by the rearrangement of the feature points.

Next, in block 148A, the image output device 28 transmits the image obtained in block 146A to, for example, a computer with which the user interacts. The transmitted image is displayed on the display of the dialogue partner.

FIG. 14 is an example of the image to be processed before processing. FIG. 15 is an example of an image obtained by performing the processing of FIG. 6 on the image of FIG. In FIG. 14, the line of sight is directed at the display and not at the camera, but in FIG. 15, the line of sight appears to be directed at the camera. Therefore, to the dialogue partner, it seems that the line of sight is directed to oneself, and a natural dialogue becomes possible.

After that, in the block 140B, a new image to be processed (second frame) is input from the camera 16 to the feature point extractor 18, and the same processing as that for the first frame is performed. However, the feature points are not extracted, and instead, the corrected feature points obtained in the block 144A are used. The processing of blocks 144B, 146B and 148B is the same as the processing of blocks 144A, 146A and 148A described above, respectively. The feature point extractor 18 may newly select and use an appropriate reference image, or may use the same reference image as the first frame. The same processing is performed for the subsequent frames. If the correlation value of Equation 6 in a certain frame is significantly lower than that of the previous frame, such as when the user blinks, it is judged that the tracking of the eye area has failed, and the frame is set as the first frame. It is handled, and a series of processes including the feature point extraction process (block 142) are performed again.

FIG. 16 is a graph showing an example of the transition of the correlation value. In FIG. 16, the correlation value between the image obtained by the process of FIG. 6 and the reference image is shown for each frame. The correlation value is obtained using Equation 6 as an evaluation function. The correlation value (A in FIG. 16) when the feature point position of the block 144A or the like is corrected is larger than the correlation value (B in FIG. 16) when the feature point position is not corrected, and the value is stable. You can see that it is doing. That is, in the case of A, it can be seen that the position correction of the feature points is almost correctly performed. As a result, the eye positions of the processed image are stable in a series of frames, and a moving image with less discomfort can be obtained.

FIG. 17 is a block diagram showing a configuration example of a computer system that realizes the line-of-sight conversion device according to the embodiment of the present invention. The computer system 80 of FIG. 17 includes a processor 82, a transmitter / receiver 84, a bus 88, a memory 92, a file storage device 94, an input device 96, and a display 98. The computer system 80 may, for example, configure a television receiver or computer that the user uses for dialogue over the communication network, or a television receiver or computer that the user uses for dialogue over the communication network. It may be built in.

The processor 82 communicates with other components via the bus 88. The transceiver 84 transmits / receives data to / from a communication network such as the Internet. The transceiver 84 may be wirelessly connected to the communication network.

The memory 92 includes, for example, a RAM (random access memory) and a ROM (read only memory), and stores data and instructions. The file storage device 94 is one or more volatile or non-volatile, non-transient, computer-readable storage media. When embodiments of the invention are realized in software, for example, microcode, assembly language code, or higher level language code may be used. The file storage device 94 stores a program described by these codes and including an instruction for realizing the function of the embodiment of the present invention. The file storage device 94 may include a RAM, a ROM, an EEPROM (electrically erasable programmable read only memory), a semiconductor memory such as a flash memory, a magnetic recording medium such as a hard disk drive, an optical recording medium, and a combination thereof.

The input device 96 may include a touch screen, a keyboard, a remote controller, a mouse, and the like. The display may include a flat panel display such as a liquid crystal display or an organic EL (electroluminescence) display.

The computer system 80 can operate as the line-of-sight conversion device 10 of FIG. The processor 82 can operate as a shape model generator 14, a feature point extractor 18, a feature point position corrector 20, an image corrector 22, and an image output device 28. The file storage device 94 can operate as the reference image storage device 12.

Each functional block in the present specification can be realized by hardware such as a circuit. Alternatively, some or all of each functional block may be implemented in software. For example, such a functional block can be realized by a processor 82 and a program running on the processor 82. In other words, each functional block described herein may be implemented in hardware, software, or any combination of hardware and software.

The above embodiments are essentially preferred examples and are not intended to limit the scope of the present invention, its applications, or its uses.

As described above, the present invention is useful for a line-of-sight conversion device, a line-of-sight conversion method, and the like.

10 Line-of-sight conversion device 12 Reference image storage device 14 Shape model generator 16 Camera 18 Feature point extractor 20 Feature point position corrector 22 Image corrector

Claims (6)

A camera that acquires an image of the photographed person's face as an image to be processed, and
A reference image storage device that stores a plurality of images of the subject's face whose line of sight is directed toward the photographing device as a reference image, and a reference image storage device.
A shape model generator that generates a shape model for the shape of the subject's eyes based on the plurality of reference images.
A feature point extractor that extracts the feature points of the eyes of the photographed person in the image to be processed, and
A feature point position corrector that corrects the position of the feature point using the shape model,
The corresponding region of the reference image is transferred to the region defined by the feature point whose position has been corrected by the feature point position corrector so that the line of sight of the photographed person is directed toward the camera. It is equipped with an image corrector that corrects the image to be processed so that it looks like.
The shape model generator is a line-of-sight conversion device that generates the shape model by principal component analysis based on the feature points of the eyes of the photographed person extracted from each of the plurality of reference images. In the line-of-sight conversion device according to claim 1,
The image corrector is a line-of-sight conversion device that transforms the shape of eyes defined by the feature points in the image to be processed into the shape of eyes in the reference image by warping. In the line-of-sight conversion device according to claim 2.
The image corrector is
The area of the eye defined by the feature point after the warping is enlarged, and in the enlarged area, the image included in the processed image is changed to the image included in the reference image as it approaches the center of the eye. A line-of-sight conversion device that superimposes an image included in the reference image on an image included in the image to be processed so as to gradually change. The image of the photographed person's face is acquired by the camera as the image to be processed, and
A plurality of images of the subject's face with the line of sight facing the direction of the photographing device are stored as reference images.
Based on the plurality of reference images, a shape model for the eye shape of the photographed person is generated.
The feature points of the eyes of the photographed person in the image to be processed are extracted.
Using the shape model, the position of the feature point is corrected.
The processing is performed so that the corresponding region of the reference image is transferred to the region defined by the feature point whose position has been corrected so that the line of sight of the photographed person appears to be directed toward the camera. Correct the target image and
Generating the shape model is a line-of-sight conversion method including generating the shape model by principal component analysis based on the feature points of the eyes of the photographed person extracted from each of the plurality of reference images. In the line-of-sight conversion method according to claim 4,
Correcting the processing target image is a line-of-sight conversion method including transforming the shape of the eyes defined by the feature points in the processing target image into the shape of the eyes in the reference image by warping. In the line-of-sight conversion method according to claim 5,
Correcting the processing target image is included in the processing target image as the area of the eye defined by the feature point after the warping is enlarged and the enlarged area approaches the center of the eye. A line-of-sight conversion method comprising superimposing an image included in the reference image on an image included in the processing target image so as to gradually change from an image to an image included in the reference image.

Images