JP3535339B2 - Interpolated image generation device and contour data generation method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an interpolation image generation apparatus, a contour data generation method, and a hidden area estimation method, and particularly to a moving image having a low frame rate, for example, a moving image received by a videophone or the like. The present invention relates to an interpolation image generation device, a contour interpolation method, and a hidden area estimation method that are suitable for use in an image processing device for converting a moving image with high resolution.

[0002]

2. Description of the Related Art Conventionally, as a technique for performing motion compensation on a moving image composed of a plurality of frames to generate a predicted image, for example, a research technology report published in 1990 by the Institute of Electronics, Information and Communication Engineers (Vol.90. PRU90-137). The "examination of motion compensation using triangular patches" by Yuichiro Nakaya and Hiroshi Harashima described in pages 9 to 16) is known.

In this document, for example, the entire reference image is divided by triangular patches, and the motion vector of each vertex of the triangular patch is obtained. From this motion vector, the coefficient of the affine transformation of the triangular patch is obtained between the reference frame and the prediction frame, and the image of the prediction frame is obtained by affine transforming the triangular patch of the reference image using the affine transformation coefficient. .

On the other hand, in recent years, research has been actively conducted on a moving image having a low frame rate such as that of a videophone to generate a moving image having a smooth motion by interpolating the image between adjacent frames. There is. In this case, the interpolated image,
That is, the method of the above document can be used as a method of predicting an interpolated image. That is, each frame representing a moving image is divided into triangular patches, and the motion vector of the apex of the triangular patch is obtained between adjacent frames. Next, the position and the motion vector of each vertex of the triangular patch in the interpolated image are calculated by linear prediction from the calculated motion vector and the position of the vertex of the triangular patch in the adjacent frame. Then, the interpolation image can be generated by obtaining the coefficients of the affine transformation from the positions of the vertices of the triangular patches of the reference image and the interpolated image, and converting the triangular patches of the reference image by the affine transformation coefficient.

[0005]

However, in the above-mentioned conventional technique, the outline of the object in the image is distorted because the entire image, that is, the entire frame is divided into triangular patches and processed in a batch. There was a problem. For example, if there are multiple objects in the image, one triangular patch may be shared by multiple objects. At this time, if the movement is different for each object,
Since a predicted image is generated between objects using motion vectors of other objects, the contour of the object may be distorted.

Therefore, in order to prevent this, it is considered that the processing should be separated for each object. However, if you try to split only one object with a triangular patch,
Because the triangle patch and the outline of the object do not match,
The triangular patch must be taken wider than the object area, and the triangular patch will span other objects. If the triangular patch is narrower than the area of the object, the part of the object that is not included in the triangular patch will be missing from the output result. If the triangle patch is made smaller and the object is finely divided, an area close to the contour of the object can be processed. However, as a result of the increase in the number of triangular patches, there is a problem that the amount of calculation increases and it takes time.

When an interpolation frame is interpolated between adjacent frames of a moving image, the objects in the image overlap each other in the previous frame, and there is a region where the object is hidden behind another object. In the current frame, the objects overlap each other. If not, there is a triangular patch in the current frame that does not exist in the previous frame. In such a case, since a triangular patch that cannot be associated with the current frame and the previous frame occurs, the motion vector of the vertex of the triangular patch cannot be obtained, and there is a problem that an interpolated image cannot be generated.

SUMMARY OF THE INVENTION It is an object of the present invention to solve the above-mentioned drawbacks of the prior art and to provide an interpolation image generating apparatus capable of effectively generating an interpolation image from a moving image in which a plurality of objects having different movements exist. To do.

Further, the present invention provides a contour data generating method which enables the area along the contour of an object to be handled without increasing the amount of calculation and can effectively predict the contour of the object in an interpolated image. The purpose is to provide.

Further, according to the present invention, in a moving image in which a plurality of objects having different motions exist in an image to generate a hidden area, the texture and shape of the hidden area are effectively estimated to generate an interpolated image. It is an object of the present invention to provide a hidden area estimation method capable of performing.

[0011]

In order to solve the above-mentioned problems, an interpolation image generation apparatus according to the present invention is an interpolation image generation apparatus for forming an interpolation frame between a plurality of frames representing a moving image. A contour data detection unit that detects contour data representing the contour shape of the object in each frame, and an object image generation unit that estimates an interpolated image of the object based on the contour data from the contour data detection unit and generates the image. And an interpolating image synthesizing unit that synthesizes the object images from the object image generating unit to generate an image of an interpolation frame,
The contour data detection unit divides the object into polygonal areas, and obtains a transformation coefficient calculation unit that obtains a transformation coefficient of shape deformation between the divided patches and a predetermined polygon serving as a reference,
The normalizing means for converting and normalizing the coordinate values of each of the divided patch and the area between the patch and the contour of the object by using the conversion coefficient calculated by the conversion coefficient calculating means, and the normalizing means. The contour distance detecting means for vertically calculating the distance from the side of the converted patch to the contour of the object, and the contour data representing the contour shape by using the value obtained by the contour distance detecting means and the coordinate value of the vertex of the patch. And contour data forming means for forming the contour data.

In this case, the transform coefficient calculating means may divide the object into patches of arbitrary triangles and obtain affine transform coefficients between the triangular patches and the reference right isosceles triangle.

Further, the normalizing means means affine-transforms the coordinate value of each point in the area between the triangular patch and the contour of the object by using the affine transformation coefficient obtained by the transform coefficient calculating means, and Normalize the coordinates.

Further, the object image generation unit includes a vertex position estimating means for estimating the coordinate values of the vertexes of the patches of the object of the interpolation image from the coordinate values of the vertexes of the patches of the objects of the adjacent frames, and the vertex position estimating means. First conversion coefficient calculation means for obtaining the conversion coefficient of the shape deformation of the patch of the interpolated image from the estimated patch of the interpolated image and a predetermined polygon serving as a reference, and the first conversion coefficient calculation means for the adjacent frames from the contour data detection means. Contour distance estimating means for estimating the distance between the patch and the contour in the interpolated image based on the distance between the patch and the contour, and the interpolated image patch estimated by the vertex position estimating means and the contour distance detecting means. The contour data represented by the contour distance is converted by the conversion coefficient obtained by the conversion coefficient calculation means to change the shape and shape of the patch and contour of the interpolated image. And a second conversion coefficient calculation means for calculating the conversion coefficient of the shape deformation from the patch of the adjacent frame and the patch of the interpolated image estimated by the vertex position estimation means. An adjacent frame is associated with each pixel of the interpolated image whose shape is converted by the conversion means using the obtained conversion coefficient, and the pixel value of the interpolated image is estimated from the pixel value of the adjacent frame based on the result. And a texture estimating means for estimating the texture of the interpolated image.

In this case, each object is divided by a triangular patch, and the first conversion coefficient calculation means
The affine transformation coefficient is obtained between the triangular patch of the interpolated image estimated by the vertex position estimation means and the reference isosceles right triangle, and the second transformation coefficient calculation means is used for the image of the adjacent frame and the interpolated image. It is advisable to obtain the affine transformation coefficient from between the triangular patches in. Further, the transforming means transforms the shape of the triangular patch and the shape region of the interpolated image using the affine transform coefficient obtained by the first transform coefficient calculating means, and the texture estimating means obtains by the second transform coefficient calculating means. It is advisable to convert the triangular patches of the adjacent frame and the triangular patches of the interpolated frame by using the affine transformation coefficient and associate them.

The interpolation image generating apparatus according to the present invention is
A hidden area complementing unit that detects whether or not the object of each frame has a hidden area hidden by another object, and complements the hidden area when there is a hidden area to generate each object image, An object image generation unit that generates each object image of an interpolation frame based on the object image complemented by the hidden area complementation unit, and an interpolation image that generates an interpolation frame by combining each object image from the object image generation means The hidden area complementing section includes a synthesizing unit and divides the object in the moving image into polygonal areas, and sets the coordinate values of the vertices of the patches of the hidden area hidden by other objects to the vertices of the patch groups of other frames. Apex coordinate value estimating means for estimating from the coordinate values of the A concealment area estimation means for estimating the circumference,
And a texture estimating means for predicting the pixel value of each point of the hidden area estimated by the hidden area estimating means from the corresponding patch of another frame.

In this case, the vertex coordinate value estimating means determines whether or not the target object has a hidden area from the coordinate values of the vertices of the patch when the object is divided into polygonal patches. Means, a reference patch detection means for detecting a patch of another frame in which the coordinate value of the patch corresponding to the hidden area is defined when the object has a hidden area, and a patch existing in the vicinity of the patch of the hidden area A conversion coefficient calculation unit that obtains a shape conversion coefficient from a patch of another frame corresponding to this, and a patch of another frame detected by the reference patch detection unit by the conversion coefficient from the conversion coefficient calculation unit are converted. Transform estimation means for estimating the coordinate values of the patch in the hidden area may be included.

Further, the vertex coordinate value estimating means defines the coordinate value of the patch corresponding to the hidden area when the patch in which the coordinate value of the vertex is defined by the conversion coefficient calculating means does not exist near the hidden area. Reference frame detection means for detecting another frame different from the frame detected by the reference patch detection means, and linear estimation means for linearly predicting the coordinate values of the vertices of the patch in the hidden area from the coordinate values of the patches of these two frames. It is good to include and.

Further, it is preferable that the transform coefficient calculating means obtains the affine transform coefficient from the patch in the vicinity of the concealed area divided into the triangular patches and the patch of the other frame corresponding thereto.

Further, the concealment area estimation means detects reference frame detection means for detecting at least two frames adjacent to the frame including the concealment area and not including the concealment area, and the frame detected by the reference frame detection means. First conversion means for obtaining a transformation coefficient of shape deformation between the patch and a predetermined polygon serving as a reference, and transforming and normalizing an area between the patch and the contour of each frame by the transformation coefficient. And a contour distance calculating means for obtaining the distance from the side of the patch to the contour after the normalization, and a distance estimation for estimating the distance from the patch to the contour in the frame including the concealed area from the value obtained by the contour distance calculating means. Means, the transformation coefficient of the shape deformation between the normalized hidden area patch and the patch of the hidden area estimated by the vertex coordinate value estimating means is obtained, and normalized using the transformation coefficient. It may comprise a second converting means for estimating the range of concealment area each point and the shape transformation of the shape regions included in the value obtained by 蔽領 area of the patch and the distance estimation unit.

In this case, the first transforming means obtains an affine transform coefficient from the patch of the frame divided into triangular patches and the reference isosceles right triangle, and the patch of each frame is obtained by the transform coefficient. The region between the contour and the contour is affine-transformed and normalized, and the second transforming means obtains an affine transform coefficient between the normalized triangular patch and the estimated triangular patch, It is preferable to use the affine transformation to calculate each point in the hidden area.

Further, the texture estimating means includes a reference frame detecting means for detecting at least two frames adjacent to the frame including the hidden area and not including the hidden area, and a patch of the frame detected by the reference frame detecting means. Conversion coefficient calculation means for calculating a conversion coefficient of shape deformation between the patch of the hidden area estimated by the hidden area estimation means,
The reference position calculating means for converting the coordinate value of the pixel in the hidden area by the conversion coefficient obtained by the conversion coefficient calculating means and associating the coordinate position of the patch of the frame detected by the reference frame detecting means with the reference position calculating means. Pixel value estimating means for estimating the pixel value of the hidden area based on the pixel values of the two frames associated with each other.

In this case, the transform coefficient calculating means obtains the affine transform coefficient from the image patch divided into the triangular patches and the estimated triangular patch, and the reference position calculating means calculates the coordinates of the pixels in the hidden area. It is advantageous to transform the value by affine transformation to obtain the coordinate value of the corresponding reference frame.

Further, the object image generation unit includes a vertex position estimating means for estimating the coordinate values of the vertexes of the patch of the object of the interpolated image from the coordinate values of the vertexes of the patch of the object of the adjacent frame, and the vertex position estimating means. The transformation coefficient of the shape deformation of the patch of the interpolated image is obtained from the estimated interpolation image patch and the reference predetermined polygon, and the transformation coefficient of the shape deformation is calculated from the reference frame patch and the reference predetermined polygon. A first conversion coefficient calculating means for calculating, a normalizing means for normalizing the patch of the reference image and a region between the patch and the contour with the conversion coefficient calculated by the conversion coefficient calculating means, and a normalizing means. The distance from the side of each patch to the contour is obtained from the area between the normalized patch and the contour, and the distance between the patch and the contour in the interpolated image is estimated from the result. The contour distance estimating means and the contour distance estimated by the contour distance estimating means and the triangular patch of the normalized interpolated image are converted by the conversion coefficient obtained by the first conversion coefficient calculation to obtain an actual frame. Transforming means for transforming the coordinate values of the triangular patch and the contour, second transforming coefficient computing means for computing the transforming coefficient of the shape deformation from the patch of the adjacent frame and the patch of the interpolated image, and the second transforming coefficient computing means. The image of the adjacent frame and each pixel of the interpolated image are associated with each other using the conversion coefficient obtained in step 1, and the pixel value of the interpolated image is estimated from the pixel value of the adjacent frame based on the result and Texture estimation means for estimating the pixel value from the pixel value of the adjacent frame and estimating the texture of the interpolated image may be included.

In this case, the first transform coefficient calculating means finds the affine transform coefficient from the patch of the image divided by the triangular patch and the reference isosceles right triangle, and the normalizing means uses the first transform coefficient. The area between the patch and the contour of the reference image is normalized by the affine transformation with the affine transformation coefficient obtained by the transformation coefficient calculating means, and the second transformation coefficient calculating means calculates the affine transformation coefficient from each triangular patch of the image. The texture estimating means may affine-transform the coordinate values of the pixels of the interpolated image using the affine transform coefficient obtained by the second transform coefficient calculating means to obtain the corresponding coordinate value of the adjacent frame.

Further, the interpolation image generating apparatus according to the present invention divides an object detection unit for detecting each object from a frame including a plurality of objects and an object detected by the object detection unit into polygonal areas. , A vertex position estimation unit that estimates the coordinate values of the vertices of the patch of the hidden region hidden by another object from the coordinate values of the vertices of the patch group of another frame, and the hidden region based on the coordinate values from the vertex position estimation unit. A hidden area estimation unit that estimates the range of the contour area, a contour data generation unit that represents each object detected by the object detection unit as a contour shape including the hidden area estimated by the hidden area estimation unit, and a hidden area complementing unit. Object image that generates each object image of the interpolation frame based on the object image that is complemented by A generating unit, characterized in that it comprises an interpolated image synthesizing unit which synthesizes each of the object image from the object image generating means for generating an interpolation frame.

In this case, the vertex position estimating unit determines whether or not the target object has a hidden area from the coordinate values of the vertices of the patch when the object is divided into polygonal patches. If the object has a hidden area, reference patch detection means for detecting a patch in another frame in which the coordinate values of the patch corresponding to the hidden area are defined, and a patch existing in the vicinity of the patch in the hidden area And a conversion coefficient calculation means for obtaining a shape conversion coefficient from a patch of another frame corresponding to the above, and a concealment area by converting the patch of another frame detected by the reference patch detection means with the conversion coefficient from the conversion coefficient calculation means. And a coordinate value estimation unit that estimates the coordinate value of the patch.

Further, the transform coefficient calculating means may find the affine transform coefficient from the patch in the vicinity of the concealed area divided into the triangular patches and the patch of another frame corresponding thereto.

Further, the concealment area estimation unit is adjacent to the frame including the concealment area and includes at least 2 that does not include the concealment area.
Reference frame detection means for detecting a number of frames; conversion coefficient calculation means for calculating a conversion coefficient of shape deformation between the patch of the frame detected by the reference frame detection means and the patch of the frame including the hidden area; Reference pixel position calculation means for calculating the position of the pixel of the reference frame corresponding to the pixel of the concealment area using the conversion coefficient obtained by the coefficient calculation means, and the corresponding concealment from the pixels of the reference frame calculated by the calculation means It is preferable to include a texture estimation unit that obtains the pixel value of the area and estimates the texture of the hidden area.

In this case, the transform coefficient calculating means may find the affine transform coefficient from the patch of the hidden area divided into triangular patches and the triangular patches of other frames corresponding thereto.

Further, the contour data generating section divides the object into polygonal areas, and obtains a transformation coefficient calculating means for obtaining a transformation coefficient of the shape deformation between the divided patches and a predetermined reference polygon. The normalizing means for converting and normalizing the coordinate values of each of the divided patch and the area between the patch and the contour of the object by using the conversion coefficient calculated by the conversion coefficient calculating means, and the normalizing means. The contour distance detecting means for vertically calculating the distance from the side of the converted patch to the contour of the object, and the contour data representing the contour shape by using the value obtained by the contour distance detecting means and the coordinate value of the vertex of the patch. It is preferable to include a contour data forming means for forming.

In this case, the transform coefficient calculating means divides the object into triangular patches and obtains affine transform coefficients between the divided triangular patches and the isosceles right triangle,
The normalizing means may affine-transform the coordinate values of the inside of the triangular patch and the region between the patch and the contour of the object with the obtained affine transformation coefficient to normalize.

Further, the object image generation unit includes a vertex position estimating means for estimating the coordinate values of the vertexes of the patch of the object of the interpolated image from the coordinate values of the vertexes of the patch of the object of the adjacent frame, and the vertex position estimating means. The transformation coefficient of the shape deformation of the patch of the interpolated image is obtained from the estimated interpolation image patch and the reference predetermined polygon, and the transformation coefficient of the shape deformation is calculated from the reference frame patch and the reference predetermined polygon. A first conversion coefficient calculation means to be obtained,
Between the patch of the reference image and the normalization means for normalizing the region between the patch and the contour with the conversion coefficient obtained by the conversion coefficient calculation means, and the patch and the contour normalized by the normalization means The contour distance estimating means for obtaining the distance from the side of each patch to the contour from the area of, and the distance between the patch and the contour in the interpolation image from the result, the patch of the adjacent frame and the patch of the interpolation image Second conversion coefficient calculation means for calculating the transformation coefficient of the shape deformation from the image, and the pixel of the adjacent frame and each pixel of the interpolated image are associated with each other using the conversion coefficient obtained by the second conversion coefficient calculation means. A texture estimation unit that estimates the pixel value of the interpolated image from the pixel value of the adjacent frame based on the result and estimates the texture of the interpolated image.

In this case, the first transform coefficient calculating means finds the affine transform coefficient from the patch of the image divided by the triangular patch and the reference isosceles right triangle, and the normalizing means uses the first transform coefficient. The area between the patch and the contour of the reference image is normalized by the affine transformation with the affine transformation coefficient obtained by the transformation coefficient calculating means, and the second transformation coefficient calculating means calculates the affine transformation coefficient from each triangular patch of the image. The texture estimating means may affine-transform the coordinate values of the pixels of the interpolated image using the affine transform coefficient obtained by the second transform coefficient calculating means to obtain the corresponding coordinate value of the adjacent frame.

On the other hand, according to the contour data forming method of the present invention, in the contour data forming method for detecting the contour shape of the object of each frame of the moving image, the object is divided into polygonal areas, and the divided patches and the reference are used. A first step of obtaining the transformation coefficient of the shape deformation with respect to the predetermined polygon, a second step of transforming the area between the side of the patch and the contour of the object using the transformation coefficient, and And a third step of vertically calculating the distance from any position on the side of the patch to the contour of the object, and using the coordinate values of the vertices of the patch and the distance from the side of the patch to the contour to determine the contour shape. Characterized by representing.

In this case, in the first step, the object is divided into triangular areas, and the affine transformation coefficient is obtained between the divided triangular patches and the reference right-angled isosceles triangle, and the second step is the first step. It is advisable to affine-transform each point of the triangular patch and each point between the patch and the contour by using the affine transformation coefficient obtained in the step of.

Further, there is provided an interpolation image generating method for generating an image of an interpolation frame based on the contour data formed by the contour data forming method according to the present invention, which is interpolated from the coordinate values of the vertices of the patches of the objects of the adjacent frames. First step of estimating the coordinate values of the vertices of the patch of the image object, and transformation of the patch shape deformation of the interpolation image from the interpolation image patch estimated in the first step and a predetermined reference polygon The second step of obtaining the coefficient and the step of generating the patch and the contour in the interpolated image based on the distance between each of the patches and the contour in the frame adjacent to the interpolation frame in the contour data formed by the above-mentioned contour data forming method The contour data represented by the third step of estimating the distance between them, the patch of the interpolated image estimated in the first step, and the contour distance estimated in the third step. Is converted by the conversion coefficient obtained in the second step to shape-convert the patch and contour of the interpolated image, the patch of the adjacent frame, and the patch of the interpolated image estimated in the first step. The fifth step of calculating the transformation coefficient of the shape deformation from the above, the adjacent frame using the transformation coefficient obtained in the fifth step and each pixel of the interpolation image subjected to the shape transformation in the fourth step are associated with each other. And a sixth step of estimating the pixel value of the interpolated image from the pixel value of the adjacent frame based on the result, and estimating the texture of the interpolated image.

In this case, it is advisable to generate an interpolated image for each object in the first to sixth steps and combine the object images to generate an interpolated frame.

The hidden area detecting method according to the present invention is
When an object in each frame of a moving image has an area hidden by another object, in the hidden area detection method for detecting the hidden area, the object is divided into polygonal patches, and the coordinates of the vertices of the patches in the hidden area are detected. A first step of estimating the value from the coordinate values of the vertices of the corresponding patch of another frame, a second step of estimating the range of the hidden area based on the coordinate value estimated in the first step, Two
And a third step of predicting the pixel value of each point in the hidden area estimated in the step of (3) from the pixel value of the corresponding patch of another frame.

In this case, in the first step, the coordinate values of the vertices of the patch in the hidden area may be estimated from the coordinate values of the corresponding vertices of another frame.

The first step is to obtain the transformation coefficient of the shape deformation from the patch existing in the vicinity of the patch in the hidden area and the patch of the other frame corresponding to the patch, and the patch of the other frame. It is preferable to include a step of converting the coordinate values of the vertices of the above with the obtained conversion coefficient, and a step of estimating the coordinate values of the vertices of the patches of the hidden area based on the result.

Further, it is advantageous that the transform coefficient in the first step is an affine transform coefficient obtained from a patch of an object divided into triangular patches.

Further, in the second step, the conversion coefficient of the shape conversion is calculated for each of the patch of the hidden area estimated in the first step and the corresponding patch of another frame and a predetermined polygon as a reference. A step of obtaining, a step of normalizing the patch of the hidden area and a patch of another frame using the conversion coefficient, a step of calculating the distance from the side to the contour in the patch of the normalized other frame, From the obtained result, the step of estimating the distance from the side of the patch normalized in the hidden area to the contour, and the transformation coefficient of the shape deformation from the normalized hidden area patch and the hidden area patch on the image are calculated. It is preferable to include a step of obtaining and a step of estimating the shape of the concealed area by transforming the shape of the patch normalized by the obtained conversion coefficient and the area between the side and the contour.

In this case, the normalization of the second step is to affine-transform each patch with the affine transformation coefficient calculated between the patch of the object divided into triangular patches and the isosceles right triangle. Good.

The shape conversion in the second step uses the affine transformation coefficient obtained from the normalized triangular patch of the concealed area and the predicted triangular patch to form the area between the normalized patch and the contour. Should be affine transformed.

Further, the third step is to obtain the transformation coefficient of the shape deformation from the coordinate values of the vertices of the patch of the hidden area and the coordinate values of the vertices of the corresponding patches of other frames, and the obtained transformation coefficient is obtained. Using the step of associating the pixels of the patch of the hidden area with the pixels of the other frame, and estimating the pixel value of the area between the patch and the contour of the hidden area based on the values of the pixels of the associated other frame .

In this case, in the third step, the affine transformation coefficient is obtained from the triangular patch of the hidden area and the triangular patch of another frame, and the coordinate value of the pixel of the hidden area is affine-transformed by the conversion coefficient. It is preferable to find the corresponding pixel position of the frame.

On the other hand, it is an interpolation image generation method for generating an interpolation frame of a moving image from an image of a frame estimated using the above-mentioned hidden area estimation method, which is interpolated from the coordinate values of the vertices of the patch of the object of the adjacent frame. The first step of estimating the coordinate values of the vertices of the patch of the object of the image, and the transformation coefficient of the shape deformation of the patch of the interpolation image is obtained from the estimated patch of the interpolation image and a predetermined polygon serving as a reference, and referred to. The second step of obtaining the transformation coefficient of the shape deformation from the frame patch and the predetermined polygon as the reference, and normalizing the patch of the reference image and the area between the patch and the contour with the obtained transformation coefficient. The distance from the side of each patch to the contour is obtained from the area between the third step and the normalized patch and the contour, and the patch and the contour in the interpolated image are obtained from the result. A fourth step of estimating the distance between them, a fifth step of calculating the transformation coefficient of the shape deformation from the patch of the adjacent frame and the patch of the interpolated image, and an image of the adjacent frame using the obtained transformation coefficient. A sixth step of associating each pixel of the interpolated image with each other, estimating the pixel value of the interpolated image from the pixel values of the adjacent frames based on the result, and estimating the texture of the interpolated image. To do.

In this case, it is preferable to generate an interpolated image for each object in the first to sixth steps and combine the object images to generate an interpolated frame.

[0050]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of an interpolation image generating apparatus, a contour data generating method and a hidden area estimating method according to the present invention will be described in detail with reference to the accompanying drawings. Figure 1
1 shows a first embodiment of an interpolation image generation apparatus to which the contour data generation method according to the present invention is applied.

The interpolation image generation apparatus according to this embodiment sequentially generates contour data representing the contour shape of each object from each frame of a moving image, and based on the contour data, generates an interpolation image of each object of the interpolation frame. It is an image generation device that generates an interpolation frame by synthesizing those images. In particular, in the present embodiment, when the contour data is generated, the object of the input frame is divided into triangular patches, the divided triangular patches are normalized into isosceles right triangles, and the normalized triangular patches are The main feature point is that contour data representing each object by the distance between the triangular patch and the contour is generated by obtaining the distance between the contour and the contour.

To explain the details, the interpolation image generating apparatus according to the present embodiment, as shown in FIG.
10, an interpolation object image generation unit 12, and a complementary image synthesis unit
14, the shape calculation unit 10 is supplied with the data of the layer image representing the effective area of the object and the data representing the vertex position of the object from the prepared vertex file, and the interpolation object image generation unit 12 The data representing the vertex position and the data of the input image are sequentially supplied to.

Explaining the details of each part, the shape calculation part
Reference numeral 10 is a contour data generation circuit that generates contour data representing the contour shape based on the layer image and the coordinate values of the vertices of the object. The effective area of the object is divided into triangular patches by combining the coordinate values of the vertices. Then, the distance to the area between the side of the patch and the contour, that is, the shape area is obtained, and contour data represented by the combination of the vertices of the triangular patch and the distance to the shape area is generated.

For example, an object is divided into a triangular patch ABD and a triangular patch BCD when the vertex of the layer image is ABCD, as shown in FIG.
Contour data represented by the combination of D's vertex coordinates and the distances from sides AB and AD to the respective contours, and triangular patches
It is generated as contour data represented by the vertex coordinates of BCD and the distances from the sides BC and CD to the respective contours. In the present embodiment, each data is normalized and formed to facilitate the subsequent processing.

More specifically, the shape calculator of this embodiment
As shown in FIG. 2, 10 is an affine transformation coefficient calculation unit 100.
The affine transformation unit 102 and the contour distance calculation unit 104 are included to normalize the triangular patches obtained by dividing the object into right-angled isosceles triangles, and calculate the coordinate values of the normalized vertices and the distance to the contour.

Specifically, the affine transformation coefficient calculation unit 100
Is a processing circuit for obtaining parameters of affine transformation when normalizing each triangular patch obtained by dividing an object, and is an arithmetic circuit for computing affine transformation coefficients based on the coordinate values of the vertices of the object from the vertex file. . For example, as shown in FIG. 5, triangular patches A0-B0-D0
Let (x1, y1), (x2, y2), (x3, y3) be the coordinates of the vertices of, and the coordinates of the vertices of the reference right-angled isosceles triangle A1-B1-C1 are (x4, y4), ( Assuming x5, y5) and (x6, y6), the affine transformation coefficients are obtained as the six coefficients a to f by the following equation (1).

[0057]

[Equation 1] The affine transformation coefficient calculation unit 100 supplies the obtained affine transformation coefficients a to f to the affine transformation unit 102.

The affine transformation unit 102 is a normalization circuit that receives the layer image data and the affine transformation coefficients a to f and normalizes the triangular patches obtained by dividing the object by affine transformation. Is a processing circuit for obtaining and outputting as coordinate values corresponding to the layer image when the coordinate values of the points inside the triangular patch and the points of the shape area are normalized. For example, the original coordinate value is (xo, yo)
Then, the converted coordinate value (xp, yp) is calculated by the following equation (2).

[0059]

[Equation 2] The affine transformation unit 102 sequentially transforms each point of the layer image by the affine transformation coefficient according to the above equation (2), and uses these coordinate values as the pixel values of the corresponding points of the layer image to sequentially contour the distance calculation unit 104. Supply to.

The contour distance calculation unit 104 includes an affine transformation unit 10
This is a processing circuit that calculates the range of the shape area of the layer image normalized based on the coordinate value data from 2, and calculates the range of the shape area by calculating the distance from the side of the triangular patch to the contour line. Circuit. For example, if the side A1-B1 goes to the contour line in the normalized triangular patch A1-B1-C1, the points are taken from A1 to B1 on the side A1-B1 at regular intervals. A perpendicular line is formed from the side A1-B1 to the contour, the length is sequentially obtained, and the contour distance is calculated. The result is supplied to the interpolation object image generation unit 12 as contour data together with the coordinate values of the vertices of the normalized triangular patch.

Returning to FIG. 1, the interpolation object image generation unit 12 receives a plurality of frames representing a moving image together with the contour data from the shape region calculation unit 10 and the vertex data from the vertex file, and the interpolated object image generation unit 12 receives the data between adjacent frames. Is an image generation circuit that generates an interpolated image of the object, and in particular, this embodiment is an image processing circuit that divides the object of each frame into triangular patches and generates an interpolated image based on the shape conversion of the triangular patches. .

More specifically, the interpolated object image generation unit 12 according to the present embodiment, as shown in FIG. 3, has a vertex position estimation unit 120, a first affine transformation coefficient calculation unit 122, a contour distance estimation unit 124, An affine transformation unit 126, a second affine transformation coefficient calculation unit 128, and a texture estimation unit 130 are included. The vertex position estimation unit 120 is a circuit that receives the vertex data of the object of each frame from the vertex file and estimates the vertex position of the triangular patch that divides the interpolated image. It is an arithmetic circuit that linearly predicts the vertex positions of the triangular patches of the object in the interpolated frames. For example, as shown in FIG. 6, a triangular patch A0-B0- in the frame at time t0.
The time between C0 and the triangular patch A2-B2-C2 at time t2
When estimating the triangular patches A1-B1-C1 of the interpolation frame of t1, from the corresponding vertices A0 = (x0, y0) and vertices A2 = (x2, y2), the vertices A1 = (x1, y1) of the interpolated image are , Is calculated by the following equations (3) and (4).

[0063]

[Formula 3] x1 = {(t1-t0) ・ x2 + (t2-t1) ・ x0} / (t2-t0) ・ ・ ・ (3) y1 = {(t1-t0) ・ y2 + (t2-t1) ・y0} / (t2-t0) (4) The vertex position estimation unit 120 similarly finds the vertex positions of the vertices B1 and C1 and other triangular patches, and calculates the result as the first affine transformation coefficient operation. It is supplied to the unit 122 and the second affine transform coefficient calculation unit 128.

The first affine transformation coefficient computing section 122 is a computing circuit for obtaining affine transformation coefficients when the triangular patch of the interpolated image from the vertex position estimating section 120 is normalized into an isosceles right triangle. 10 conversion coefficient calculator 10
Similar to 0, using the above equation (1), the affine transformation coefficient a ~ between the triangular patch of the interpolation image and the reference right-angled isosceles is used.
This is a processing circuit for obtaining f. Obtained affine transformation coefficient a
To f are supplied to the affine transformation unit 126.

The contour distance estimator 124 includes a shape calculator 10
Is a processing circuit that receives the contour data from the contour image and estimates the distance between the side of the triangular patch and the contour in the interpolated image, and interpolates between them from the normalized distances of the shape regions in two adjacent frames. It is an arithmetic circuit for obtaining the distance of the normalized shape region in the frame by linear prediction. For example, as shown in FIG. 7, the normalized triangle patch A01-B01-D01 at time t0 measures the distance from the point x on the side A01-D01.
Let h0 be the normalized triangular patch A21-B21-D21 at time t2.
Then, if the distance from the point x on the side A21-D21 is h2, the time is
Normalized triangular patch of t1 A11-B11-D11 sides A11-D11
The distance h1 from the upper point x to the contour is calculated by the following equation (5).

[0066]

[Equation 4] h1 = {(t1-t0) ・ h2 + (t2-t1) ・ h0} / (t2-t0) (5) The contour distance estimation unit 124 uses the above equation (5) to interpolate the image. The distance from the normalized triangle patch to the contour is sequentially obtained, and together with the result, the coordinate value of the vertex position of the normalized triangular patch of the interpolated image is supplied to the affine transformation unit 126.

The affine transformation unit 126 includes a contour distance estimation unit 12
The normalized triangular patch of the interpolated image from 4 and each value of the shape region are converted by the affine transformation coefficients a to f from the first affine transformation coefficient unit 122, and the triangular patch in the actual interpolated image and This is a processing circuit for returning to the value of the shape region, and is an arithmetic circuit for performing conversion using the equation (2) similarly to the affine transformation unit 102 of the shape calculation unit 10. For example,
In Figure 7, the normalized triangular patch A11-B11-D1 of the interpolated image
Each point in 1 and the shape region is converted into a triangular patch A1-B1-D1 in the actual interpolated image and a point in the shape region. The converted result is supplied to the texture estimation unit 130.

On the other hand, in FIG. 3, the second affine transformation coefficient computing unit 128 performs shape transformation from the vertex positions of the triangular patches of the interpolated image from the vertex position estimating unit 120 and the vertex positions of the triangular patches of the adjacent frames. This is an arithmetic circuit for obtaining the affine transformation coefficients which become the coefficients, and the affine transformation coefficients a to f are obtained by using the equation (1) similarly to the affine transformation coefficient operation unit 100 of the shape calculation unit 10. For example, in FIG. 6, the shape conversion coefficient between these is calculated from the triangular patch A1-B1-C1 of the interpolated image at time t1 and the triangular patch A0-B0-C0 of the previous frame at time t0, and the triangular patch of the interpolated image is further calculated. From A1-B1-C1 and the triangular patch A2-B2-C2 of the current frame at time t2, the shape conversion coefficient between them is obtained. The obtained affine transformation coefficient is supplied to the texture estimation unit 130.

The texture estimating unit 130 is a processing circuit that estimates the pixel value of the interpolated image by associating the triangular patch of the interpolated image with the triangular patch of the adjacent frame, and outputs from the second affine transformation coefficient operation unit 128. Convert the coordinates of each point of the triangular patch and shape area of the interpolated image to the coordinates of the corresponding frame using each conversion coefficient, and linearly estimate the pixel value of the interpolated image from the pixel value of the resulting coordinates It is an arithmetic circuit. For example, as shown in FIG.
Using the affine transformation coefficient between the triangular patch A1-B1-C1 at time t1 and the triangular patch A0-B0-C0 at time t1 obtained by the affine transformation coefficient calculator 128 of Pixel
The position of the pixel P0 in the triangular patch A0-B0-C0 corresponding to P1 is obtained. Similarly, pixel P1 in triangular patch A1-B1-C1 at time t1
The position of the pixel P2 in the triangular patch A2-B2-C2 at time t2 corresponding to is calculated. From these results, from the pixel value Q0 of the pixel P0 of the previous frame and the pixel value Q2 of the pixel P2 of the current frame, the following equation (6)
Then, the pixel value Q1 of the pixel P1 of the interpolated image is obtained.

[0070]

[Equation 5] Q1 = {(t1-t0) ・ Q0 + (t2-t1) ・ Q2} / (t2-t0) (6) The texture estimation unit 130 calculates all pixels in the triangular patch and in the shape region. The values are sequentially estimated to generate the object image of the interpolation frame, and the object image is supplied to the interpolation image synthesizing unit 14.

The interpolated image synthesizing unit 14 is an image processing circuit for synthesizing and outputting the object from the interpolated object image generating unit 12 into one frame, and sequentially maps the interpolated images from those spatially rearward. And a frame generation circuit for forming an interpolation frame.

The operation of the interpolation image generating apparatus according to the present embodiment having the above configuration will be described together with the contour data generating method according to the present embodiment. In the case of this embodiment, for example,
As shown in FIG. 9, when the ball L on the right side of the screen at the frame M at the time t0 rotates 180 degrees clockwise and moves to the left side of the screen at the frame N at the time t2, those frames are moved. An example will be described in which an interpolation frame at an intermediate time t1 is generated.

In the operation state, first, when the frame M at time t0 is input, the layer image showing the effective area of the ball L and the coordinate value of the vertex data ABCD of the ball L are supplied to the shape calculation unit 10. . As a result, the shape calculation unit 10 sets the ball L to the triangular patch ABD and the triangular patch BCD.
Then, the calculation of the contour data is started.

First, when the vertex data ABCD is supplied to the affine transformation coefficient calculation unit 100, the affine transformation coefficient calculation unit 10
0 is the affine transformation coefficient a to f obtained from the vertex coordinates of the triangular patch ABD and the vertex coordinates of the reference right-angled isosceles triangle in the above equation (1), and the result is obtained by the affine transformation unit 102.
Supply to. As a result, in the affine transformation unit 102 that has received the affine transformation coefficients a to f, the coordinate values of the points of the triangular patch ABD and its shape region are affine transformed and normalized from the data of the layer image by the above equation (2). ,
The result is supplied to the contour distance calculation unit 104.

Next, the contour distance calculation unit 104 that receives the coordinate values of the normalized triangular patch ABD and its shape area.
Then, the distance from the side of the triangular patch to the contour is sequentially calculated, and the result is supplied to the interpolation object image generation unit 12 as the contour data together with the vertex coordinates of the triangular patch.

When the calculation of the triangular patch ABD is completed, the shape calculation unit 10 obtains the normalization of the triangular patch BCD and the contour distance of the shape area and outputs the contour data to the interpolation object image generation unit 12 as in the above. Supply. As a result, the contour data of the ball L of the frame at time t0 is supplied to the interpolation object image generation unit 12.

Next, when the shape calculation unit 10 is supplied with the vertex data A'B'C'D 'of the ball L in the frame at time t2 and the layer image thereof, the shape calculation unit 10 receives the same data as described above. The ball L is divided into a triangular patch A'B'D 'and a triangular patch B'C'D', which are normalized into an isosceles right triangle, and the normalized coordinates of the triangular patch and the contour distance of the shape region are calculated. The contour data represented by is supplied to the interpolation object image generation unit 12.

Next, in the interpolated object image generation unit 12 that has received the contour data, first, in the vertex position estimation unit 120, the vertex data ABCD of the ball L in the frame at time t0 and the frame at time t2 from the vertex file. Vertex data of A'B'C '
Read D '. Next, the vertex position estimation unit 120, like the shape calculation unit 10, calculates the ball L at the respective times t0 and t2.
Is divided by triangular patches, and the vertex positions of the triangular patches A "B" C "D" of the ball L in the interpolation frame at the time t1 are divided by the above equations (3), (4 ), The result is calculated based on the first affine transformation coefficient calculation unit 122 and the second affine transformation coefficient calculation unit 122.
To the affine transformation coefficient calculation unit 128.

Next, in the first affine transformation coefficient calculation unit 122 which has received the vertex coordinates of the triangular patches of the interpolated image, as with the affine transformation coefficient calculation unit 100 of the shape calculation unit 10, each triangular patch and reference The affine transformation coefficient is obtained between the quadrature and the isosceles right triangle, and the results are supplied to the affine transformation unit 126.

On the other hand, the contour distance estimation unit 124 is supplied with contour data at times t0 and t2 from the shape calculation unit 10, and based on this, the distance between each triangular patch and the contour in the interpolated image is described above. It is calculated by equation (5). The result is supplied to the affine transformation unit 126. As a result, the affine transformation unit 126 transforms the coordinates of each point of the triangular patch and the shape area in the normalized interpolated image using the affine transformation coefficient from the first affine transformation coefficient calculation unit 122, and Convert to coordinates of triangular patch and shape area in image. The result is sequentially supplied to the texture estimation unit 130.

On the other hand, the second affine transform coefficient operation unit 128
Are supplied with the vertex coordinates of the triangular patch in the interpolated image at time t1 estimated by the vertex position estimation unit 120 and the vertex coordinates of the triangular patch at times t0 and t2, respectively. The affine transformation coefficient between the patch and the affine transformation coefficient between the triangular patch of the interpolated image and the triangular patch at time t2 are obtained, and these are supplied to the texture estimation unit 130. As a result, the texture estimation unit 130 affine-transforms the coordinates of the triangular patch of the interpolated image with each affine transformation coefficient, and associates each pixel position of the interpolated image with the pixel position at time t0, t2.

Next, the texture estimating section 130 sequentially finds the corresponding pixel value of the interpolated image from the associated pixel values at the times t0 and t2 by the above-mentioned equation (6), and calculates the ball L of the interpolated image.
For the texture of. As a result, as shown in FIG.
An object image of the ball L in the interpolation frame at time t1 is generated and supplied to the interpolation image synthesizing unit 14. In this case, the ball L is depicted as rotating 90 degrees clockwise from the image at time t0 and moving slightly to the left.

Next, the interpolated image synthesizing unit 14, which has received the object image from the interpolated object image generation unit 12, maps the object images sequentially into one frame from the spatially rearward one to form an interpolated frame. Form and output the result. In the case of the present embodiment, the image of the ball L generated by the interpolation object image generation unit 12 is combined with the background and output.

As described above, according to the interpolation image generating apparatus and the contour data generating method of the present embodiment, the contour shape is represented by the data of the triangular patch and the contour distance of the shape area, so that each object is represented. It enables handling along the contour, does not divide the object finely, and does not have to make the triangular patch wide around the object, so there is very little distortion of the contour of the generated interpolation image or the image around it. can do.

Further, by representing the contour shape by the data of the triangular patch and the distance of the shape region, the contour shape and texture of the object of the interpolated image can be estimated by a simple calculation using linear prediction. You can

Next, FIG. 10 shows a second embodiment of the interpolation image generating apparatus according to the present invention. The interpolation image generation apparatus according to the present embodiment estimates the image of the hidden area when there is an object having a region hidden by another object in the moving image, and calculates the interpolation frame based on the estimated image. This is an image generation device that forms an interpolation frame by accurately generating an interpolation image for each object and synthesizing those images. In particular, in this embodiment,
When estimating the concealment area, the object of the input frame is divided by triangular patches, and the coordinate values of the vertices of the divided triangular patches are estimated from the corresponding patch of another frame or a patch in the vicinity thereof, and the concealment area The main feature point is that the texture of the patch is estimated from the corresponding triangular patch of another frame.

In FIG. 10, the same parts as those in the above embodiment are designated by the same reference numerals, and the description thereof will be omitted as much as possible.
Hereinafter, only different parts will be described in detail.

Specifically, as shown in FIG. 10, the interpolation image generation apparatus according to the present embodiment includes a hidden area interpolation section 20, an interpolated object image generation section 30, and an interpolated image synthesis section 14, and includes a hidden area. The interpolation unit 20 is supplied with the layer image of each object, the vertex data from the vertex file, and the input image data.

More specifically, the hidden area interpolating unit 20 detects whether or not the object of each frame has a hidden area hidden by another object, and interpolates the hidden area when there is a hidden area. This is an image processing circuit for generating respective object images by means of this embodiment, and in the present embodiment, it includes a vertex coordinate value estimation section 22, a hidden area estimation section 24, and a texture estimation section 26.

The vertex coordinate value estimating unit 22 divides the object in the moving image into triangular patches based on the layer image and the vertex data for each object, and detects the hidden area from the coordinate values of the triangular patches. Is an estimation circuit that estimates the coordinate value of the vertex of the triangular patch from the coordinate value of the vertex of the patch group of another frame when a hidden area hidden by another object is detected. For example, as shown in FIG. 11, the vertex coordinate value estimation unit 22 of the present embodiment estimates the estimation unit 22.
0, the reference patch detection unit 222, the affine transformation coefficient calculation unit 224, the transformation estimation unit 226, and the reference frame detection unit 228.
And a linear estimation unit 230.

The estimation determination unit 220 is a determination circuit that determines whether or not each vertex of the triangular patch is included in the hidden area and whether or not the coordinate value of the vertex needs to be estimated. It is a processing circuit that performs processing separately depending on whether or not the vertex coordinates of are defined.
For example, when the vertex coordinate value from the vertex file is defined, but it is determined that the vertex overlaps another object and is outside the range of the effective area in the layer image. In this case, the vertex coordinate value is estimated. However, it is determined that the triangular patch is included in the hidden area and a signal indicating this is supplied to the reference patch detection unit 222.

On the other hand, when a vertex coordinate value from the vertex file is not defined, for example, a vertex coordinate value represented by a value such as (-9999, -9999) in this embodiment, is given. In this case, It is determined that the vertex position of the triangular patch is clearly hidden by another object, and it is necessary to estimate the vertex coordinate value, and the signal indicating the result is referred to the reference patch detection unit 22.
Supply to 2. For example, as shown in FIG. 18, time t10
If part of the lower center of the object is hidden by another object in the frame of the triangle patch C1-G1-E1
It is determined that the vertex E1 of is undefined, and the result is output.

The reference patch detection unit 222 includes an estimation judgment unit 220.
It is a detection circuit that detects a reference patch for estimating the vertex coordinate value of the triangular patch of the concealed area based on the determination result of, and the vertex of the frame for which it is determined that the coordinate value needs to be estimated in another frame. A frame in which the coordinate values of the vertices are defined is detected, and three vertices of the triangular patch are defined in each frame, and a triangular patch close to the vertex whose coordinate value is estimated is detected. For example, as shown in Figure 18, the vertices of triangular patch C1-G1-E1 in the frame at time t10
When estimating the coordinate value of E1, a frame at time t0 in which the corresponding vertex E0 is defined is detected, and at time t10, a triangular patch B1-in which three vertices are defined near the vertex E1. C
1-G1 is detected, the triangular patch B0-C0-G0 corresponding to the frame at time t0 is further detected, and the result is supplied to the affine transform coefficient calculation unit 224. Further, when a frame including a triangular patch with three vertices defined cannot be detected, a signal instructing the result to the affine transformation coefficient calculation unit 224 and the reference frame detection unit 228 is output.

The affine transformation coefficient calculation unit 224 is a calculation circuit for obtaining the transformation coefficient of the shape deformation between the triangular patches detected by the reference patch detection unit 222, and the affine transformation coefficient calculation unit 224 calculates the transformation coefficient of the vertices in the vicinity of the vertices. For a triangular patch, affine transformation coefficients are obtained from the three vertices at the time of estimating the vertex coordinate values and the three vertices of another frame. The arithmetic expression is the same as the expression (1) described in the above embodiment, and the six coefficients a to f are obtained in the same manner as above, and the result is, for example, the coordinate value of the corresponding point E0 of the reference patch. It is also supplied to the conversion estimation unit 226.

The transform estimating unit 226 uses the coordinate values supplied from the affine transform coefficient computing unit 224 as the affine transform coefficients a to f.
Is a processing circuit for estimating the coordinate values of the vertices in which the triangular patches of the hidden area are not defined by converting the coordinate values of the corresponding points by using Equation (2) as in the above embodiment. And an arithmetic circuit for obtaining the value as the coordinate value of the vertex of the triangular patch in the hidden area. The obtained results are supplied to the hidden area estimation unit 24 and the interpolated object image generation unit 30, respectively.

On the other hand, the reference frame detection unit 228 is a detection circuit for detecting two frames at other times whose coordinate values are defined in the vertex file for the vertices of the hidden area for which the coordinate values are estimated. Then, the reference patch detection unit 222
It works only when the reference patch is not detected in. For example, in FIG. 15, a triangular patch B1 near the vertex E1 at time t10
-When C1-G1 cannot be detected, or when triangular patch B0-C0-G0 cannot be detected at time t0, the frame at time t0 including point E0 defined corresponding to vertex E1 and the point E2 similarly.
The frame at time t20 including is detected as a reference frame, and the result is supplied to the linear estimation unit 230.

The linear estimator 230 includes a reference frame detector 22.
This is an arithmetic circuit that linearly predicts the coordinate values of the vertices of triangular patches in the hidden area whose coordinate values are undefined based on the coordinate values of the vertices in the frame detected in 8. For example, in Figure 15, the coordinate value of vertex E0 at time t0 is (x4, y4), and the vertex at time t20 is
Assuming that the coordinate value of E2 is (x5, y5), the coordinate value (x6, y6) of the vertex E1 of the hidden area at time t10 can be calculated by the following equation (7).

[0098]

[Equation 6] x6 = {(t10-t0) ・ x5 + (t20-t1) ・ x4} / (t20-t0) ・ ・ ・ (7) y6 = {(t10-t0) ・ y5 + (t20-t1) ・y4} / (t20-t0) (8) The linear estimation unit 230 operates only when the reference patch detection unit 222 cannot detect the reference patch, like the reference frame detection unit 228, and the result is the hidden area. It is supplied to the estimation unit 24 and the interpolation object image generation unit 30.

The concealment area estimation unit 24 uses the vertex coordinate value estimation unit 22.
It is a processing circuit that estimates the range of the hidden area based on the coordinate values estimated in step 1. In the present embodiment, for example, as shown in FIG. 12, a reference frame detection section 240, a first affine transformation section 242, Contour distance calculation unit 244 and contour distance estimation unit 24
6 and a second affine transformation unit 248.

The reference frame detection unit 240 determines at least two frames adjacent to the frame including the hidden area based on the coordinate values from the vertex coordinate value estimation unit 22 and the layer image and including the triangular patch without the hidden area. A detection circuit for detecting, for example, in FIG.
When the coordinate value of the vertex E1 of the frame is supplied, the triangular patch corresponding to the triangular patch C1-E1-G1 detects the frame at time t0 and the frame at time t20 that do not include the concealed area. The result is supplied to the first affine transformation unit 242.

The first affine transformation unit 242 is a normalization circuit that normalizes each triangular patch of the frame detected by the reference frame detection unit 240, and detects the detected triangular patch and the reference right isosceles triangle. The affine transformation coefficient is obtained between the two, and the triangular patch and the shape region of each frame are affine transformed and normalized by the transformation coefficient. The result is supplied to the contour distance calculation unit 244.

The contour distance calculation unit 244 is a calculation circuit for calculating the distances from the sides of the triangular patches of the reference frame normalized by the first affine transformation unit 242 and the sides of the triangular patches in the shape area to the contour. For example, the distance to the contour is calculated at a predetermined position on the side of the triangular patch toward the contour. The results are supplied to the contour distance estimation unit 246.

The contour distance estimator 246 includes a contour distance calculator 24.
4 is a circuit for estimating the distance from the triangular patch to the contour in the frame including the concealed area based on the calculation result from 4, and in the case of the present embodiment, the normalized triangular patch is used in the same manner as the above embodiment. The contour distance is linearly predicted by the same formula as formula (5). The calculated contour distance in the hidden area is supplied to the second affine transformation unit 248 together with the coordinate value of the triangular patch.

The second affine transformation unit 248 finds the affine transformation coefficient between the normalized triangular patch of the hidden area and the triangular patch of the hidden area estimated by the vertex coordinate value estimation unit 240, and transforms it. This is an arithmetic circuit that estimates the range of the concealment area by affine-transforming the triangular patches of the concealment area that are normalized using the coefficients and the points of the shape area that are included in the values obtained by the contour distance estimation unit 246. The estimated result is the texture estimation unit 26 and the interpolation object image generation unit 30.
Is supplied to.

The texture estimating unit 26 uses the hidden area estimating unit 24.
It is a processing circuit that predicts the pixel value of each point in the range of the concealed area estimated from the corresponding patch of another frame to obtain the texture of the concealed area. In the present embodiment, for example, in FIG. As shown, the reference frame detector
260, an affine transformation coefficient calculation unit 262, a reference position calculation unit 264, and a pixel value estimation unit 266.

The reference frame detection unit 260 determines that at least 2 frames that do not include a concealment area are adjacent to a frame that includes a concealment area.
This is a detection circuit that detects one frame, and
The frame at t0 and the frame at time t20 are detected respectively. The result is supplied to the affine transform coefficient calculation unit 262.

The affine transformation coefficient calculation unit 262 calculates the affine transformation coefficient between the triangular patch of each frame detected by the reference frame detection unit 260 and the triangular patch of the hidden area estimated by the hidden area estimation unit 24. This is a calculation circuit to be obtained, and the affine transformation coefficient is calculated using the equation (1) as in the above embodiment. The result is supplied to the reference position calculation unit 264.

The reference position calculation unit 264 is a conversion circuit that affine-transforms the coordinate value of each pixel in the concealed area using the affine transformation coefficient obtained by the affine transformation coefficient calculation unit 262, and the reference frame detection unit 260 Is a detection circuit for affine-transforming each triangular patch of the detected frame to detect the pixel position of each frame corresponding to the pixel of the hidden area. Those results are supplied to the pixel value estimation unit 266.

The pixel value estimation unit 266 is a reference position calculation unit 264.
It is a processing circuit for estimating the pixel value of the concealed area based on the pixel values of the two frames associated with each other in Equation (6) used in the texture estimating unit 130 of the interpolation object image generating unit 12 in the above embodiment. ) Is an arithmetic circuit that linearly estimates the pixel value using the same formula as The calculation result is supplied to the interpolation object image generation unit 30.

As shown in FIG. 14, the interpolation object image generation unit 30 according to the present embodiment has a vertex position estimation unit 300, a first affine transformation coefficient calculation unit 302, a first affine transformation unit 304, and a contour. The distance estimation unit 306, the second affine transformation unit 308, the second affine transformation coefficient calculation unit 310, and the pixel value estimation unit 312 are included. The interpolation object image generation unit 30 of this embodiment basically has the same function as the interpolation object image generation unit 12 of the above embodiments. However, since the input / output relationship of each part is different from the above, the description will be made focusing on these points and simplifying as much as possible.

The vertex position estimation unit 300 is an arithmetic circuit for linearly estimating the coordinate values of the vertices of the triangular patches of the object of the interpolated image from the coordinate values of the vertices of the triangular patches of the objects of the adjacent frames. Area interpolator 20
The vertex data of the object in each frame is received via the vertex coordinate value estimation unit 22 of the, and the coordinate value of each vertex of the triangular patch of the interpolation image is calculated based on the vertex data including the estimated coordinate value of the hidden area. Ask. The result is
It is supplied to the first affine transformation coefficient calculation unit 302 and the second affine transformation coefficient calculation unit 310.

The first affine transformation coefficient calculation unit 302 calculates the affine transformation coefficient from the triangular patch of the interpolated image estimated by the vertex position estimation unit 300 and the triangular patches of the adjacent frames and the reference right isosceles triangle. Is an arithmetic circuit for obtaining The results are supplied to the first affine transformation unit 304 and the second affine transformation unit 308, respectively.

The first affine transformation unit 304 is an arithmetic circuit that normalizes the triangular patch and the shape area of the reference image using the affine transformation coefficient from the first affine transformation coefficient operation 302, and is a hidden area in the reference image. If there is, the normalized triangular patch from the concealment area estimation unit 24 and the value of the shape area are interpolated and output. The result is supplied to the contour distance estimation unit 306.

The contour distance estimation unit 306 obtains the distances from the sides of each triangular patch to the contour from the values of the normalized triangular patch and the shape area from the first affine transformation unit, and from the result, the interpolated image is obtained. It is an arithmetic circuit that estimates the distance between the triangular patch and the contour. The result is supplied to the second affine transformation unit 308 together with the coordinate value of the triangular patch of the interpolated image.

The second affine transformation unit 308 affine-transforms the coordinate values of the triangular patch and the shape region of the normalized interpolated image using the affine transformation coefficient from the first affine transformation coefficient operation unit 302, This is an arithmetic circuit that restores the coordinate values of the triangular patch and the shape area of the actual image.
The result is supplied to the pixel value estimation unit 312.

On the other hand, the second affine transformation coefficient calculation unit 310
Is an arithmetic circuit that calculates an affine transformation coefficient from the triangular patch of the interpolated image estimated by the vertex position estimation unit 300 and the triangular patch of the adjacent frame, and the result is the pixel value estimation unit.
Supplied to the 312.

The pixel value estimation unit 312 associates the image of the adjacent frame with each pixel of the interpolated image using the affine transform coefficient from the second affine transform coefficient operation unit, and based on the result, the interpolated image. Is a calculation circuit for estimating the pixel value of the pixel value from the pixel value of the adjacent frame, and if the reference frame has a hidden area, the pixel value of the interpolated image using the pixel value estimated by the texture estimating section 26 of the hidden area interpolating section 20. Estimate the value. From the result, the object images of the interpolation frames are respectively generated, and the respective object images are supplied to the interpolation image synthesizing unit 14.

The interpolation image synthesizing unit 14 sets the object from the interpolation object image generating unit 30 to 1 as in the above embodiment.
An image processing circuit that synthesizes and outputs a single frame,
This is a frame generation circuit that sequentially maps the interpolated images from those spatially rearward to form an interpolated frame.

The operation of the interpolation image generating apparatus according to the present embodiment having the above configuration will be described together with the hidden area estimation method according to the present embodiment. In the case of this embodiment, for example,
As shown in FIG. 15, a moving image in which the ball L on the left side of the screen rolls to the right side across the front of a rectangular object K such as a calendar will be described as an example.

In the operating state, first, when the frame at time t0 is supplied to the hidden area interpolation unit 20, the vertex coordinate value estimation unit 220 converts the layer image of each object and the vertex data of the vertex file into each object. It is detected whether or not there is a hidden area. In this case, since the object has no hidden area in the frame at time t0, each data is supplied to the interpolated object image generation unit 30 as it is through the hidden area interpolation unit 20.

Next, when the frame at time t10 is supplied to the concealment area interpolating unit 20, the apex coordinate value estimating unit 22 detects that an undefined vertex E1 is included in the lower part of the calendar K and conceals the concealment area. The interpolation operation of the hidden area in the area interpolation unit 20 starts. First, the estimation and determination unit that has detected the vertices of the hidden area
220 supplies a signal indicating that it is necessary to estimate the coordinate value of the vertex E1 to the reference patch detection unit 222.

As a result, the reference patch detection unit 222 searches for a frame including the defined vertex corresponding to the vertex E1 and detects a frame at time t0 including the defined vertex E0. Next, the reference patch detection unit 222 detects the triangular patch B1-C1-G1 including the defined three vertices near the vertex E1 in the frame at time t10, and together with this, the triangular patch B1-C1 in the frame at time t0. -Triangular patches B0-C0-G0 in which three vertices corresponding to -G1 are defined are detected, and their coordinate values are supplied to the affine transformation coefficient calculation unit 224. At this time, when the defined triangular patch B1-C1-G1 or triangular patch B0-C0-G0 cannot be detected, a signal indicating that fact is supplied to the reference frame detection unit 228. In this embodiment, since the reference patch can be detected, the reference frame detection unit 228 and the linear estimation unit 23
0 does not work. When operating, the reference frame detection unit 228 waits for the input of the frame at time t20,
Frame at time t0 including vertex E0 and time t20 including vertex E2
Frames are detected and their vertex coordinate values are supplied to the linear estimation unit 230. As a result, the linear estimation unit 230
The coordinate value of the vertex E1 is obtained by linear estimation from the coordinate values of E0 and E2, and the result is supplied to the hidden area estimation unit 24 and the interpolation object image generation unit 30.

On the other hand, in the case of this embodiment, the triangular patches B0-C0-
The affine transformation coefficient operation unit 224 that receives G0, B1-C1-G1 obtains the affine transformation coefficient from these triangular patches, and supplies the result to the transformation estimation unit 226. As a result, the transform estimation unit 226 that has received the affine transform coefficient affine transforms the coordinate value of the vertex E0 of the frame at time t0,
The value is supplied to the hidden area estimation unit 24 and the interpolated object image generation unit 30 as the coordinate value of the vertex E1.

Next, when the reference frame detection unit 240 receives the coordinate value of the estimated vertex E1 of the hidden region in the hidden area estimation unit 24, first, the frame at time t0 including the vertex E0 corresponding to the vertex E1 is received. Is detected and then the frame at time t20 is supplied, this is detected and the result is supplied to the first affine transformation unit 242. At this time, in the vertex position detection unit 22, since the object at the time t20 has no hidden area, the respective data are supplied to the hidden area estimation unit 24 and the interpolated object image generation unit 30 as they are.

Next, the first affine transformation unit 242 which has received the reference frame obtains affine transformation coefficients from the triangular patch at time t0 and the reference right-angled isosceles triangle, and normalizes each triangular patch and its shape region. Similarly, the affine transformation coefficient is found from the triangular patch at time t20 and the reference right-angled isosceles triangle, and each triangular patch and its shape region are normalized. The results are sequentially supplied to the contour distance calculation unit 244. The contour distance calculation unit 244, which has received the normalized triangular patch of the reference frame and the coordinate values of the shape region, calculates the distance to the contour at each triangular patch, and the result is calculated by the contour distance estimation unit.
Supply to 246.

As a result, the contour distance estimating unit 246
The contour distances of the normalized triangular patches in the concealment area are estimated by linear prediction from the contour distances of the normalized triangular patches of t0 and t20, and the result is calculated by the second affine transformation unit 248. Supply to. The second affine transformation unit 248 obtains an affine transformation coefficient from the triangular patch of the concealed area and the normalized triangular patch from the vertex coordinate value estimation unit 22 and uses the transformation coefficient to determine the coordinates of the triangular patch and its shape area. Is affine transformed to estimate the range of the concealed area in the actual image, and the result is supplied to the texture estimation unit 26 and the interpolation object image generation unit 30.

Next, in the texture estimating unit 26 which has received the result from the hidden area estimating unit 24, first, the reference frame detecting unit
At 260, the frame at time t0 and the frame at time t20 are detected, and triangular patches corresponding to these concealed areas are supplied to the affine transform coefficient calculation unit 262. As a result, in the affine transformation coefficient calculation unit 262, the triangular patch of the hidden area and the time
The affine transformation coefficient is calculated between the triangular patch corresponding to t0, the affine transformation coefficient is calculated between the triangular patch in the hidden area and the triangular patch at time t20, and these are supplied to the reference position calculation unit 264. To do.

Next, the reference position calculation unit 264 that has received each affine transformation coefficient uses the affine transformation coefficient obtained between the triangular patch of the concealed area and the pixel position of the shape area of the triangular patch at time t0. Then, the triangular patch of the concealed area and its shape area are associated with the pixel positions of the triangular patch and shape area at time t0. Similarly, the triangular patch of the concealment area and the pixel position of the shape area thereof are converted by the affine transformation coefficient obtained between the triangular patch of time t20 and the triangular patch of the concealment area and the time t20.
The triangular patch of and the pixel position of the shape area are associated with each other. As a result, the triangular patch in the hidden area and the triangular patches at times t0 and t20 corresponding to the pixel positions in the shape area and the shape area are associated with each other, and the result is supplied to the pixel value estimation unit 266.

As a result, the pixel value estimation unit 266 determines that the time t
Each pixel value of the hidden area is linearly estimated from the value of the corresponding pixel of 0, t20, and the result is supplied to the interpolation object image generation unit 30.

In the following, the interpolation object image generation unit 30 generates an interpolation image at time t5, for example, from the object at time t0 and the object at time t10 obtained by interpolating the hidden area, as in the above embodiment. The interpolated image at time t15 is generated from the object at time t10 and the object at time t20 in which the hidden area is interpolated, and the result is supplied to the interpolated image synthesis unit 14.

As a result, the interpolation image synthesizing unit 14 sequentially maps the object images spatially rearward to one frame to form an interpolation frame, and outputs the result. In the case of the present embodiment, the image of the ball L is slightly overlapped and output near the left end or the right end of the calendar K and output.

As described above, according to the interpolated image generation apparatus and the hidden area estimation method of this embodiment, the vertex coordinate value estimation section 22 of the hidden area interpolation section 20 detects the vertex coordinates of the hidden area, and the hidden area is detected. Since the range of the area is estimated by the hidden area estimation unit 24, and the texture of the hidden area is further estimated by the texture estimation unit 26, it is possible to effectively generate the interpolated image even when the object image includes the hidden area. it can.

FIG. 16 shows a third embodiment of the interpolation image generating apparatus according to the present invention. The interpolation image generation apparatus according to the present embodiment, in an image having a plurality of objects having different movements in a moving image, when there is an object having an area hidden in them, interpolates the hidden area,
The image processing apparatus also generates contour data of all objects including the interpolated object, and effectively generates an interpolated image based on the contour data. In particular, the present embodiment is different from the second embodiment in that it includes an object detection unit 40 that detects the effective areas of a plurality of objects in advance before estimating the concealment area, and after estimating the concealment area. The point is that it includes a shape region calculation unit 46 similar to that of the first embodiment for generating the contour data of each object.

The details will be described. The interpolation image generating apparatus of the present embodiment, as shown in FIG.
40, a vertex position estimation unit 42, a hidden region estimation unit 44, a shape region calculation unit 46, and an interpolation object image generation unit 48.
And an interpolated image synthesizing unit 14, and an object detecting unit 40
Is supplied with the layer image of each object and the image of the input frame, and the vertex position estimation unit 42 is supplied with the layer image and the data of the vertex file.
An image of the input frame and the outputs from the object detection unit 40 and the vertex position estimation unit 42 are supplied to 44, respectively.

The object detection unit 40 is an image detection circuit which cuts out the image of the effective frame of each object from the image of the input frame based on the layer image showing each object. For example, if the image of the input frame includes three images of a calendar K, a ball L, and a train M, as shown in FIG.
Images are provided as shown in FIGS. The object detection unit 40 cuts out an image of, for example, the effective area of the calendar K as shown in FIG. 24 from these, and outputs the result.

The vertex position estimating unit 42 includes substantially the same function as the vertex coordinate value estimating unit 22 of the second embodiment for detecting the hidden area of the object and detecting the vertex position of the undefined triangular patch. Circuit. For example, in the present embodiment, as shown in FIG. 17, an estimation judgment unit 220, a reference patch detection unit 222, an affine transformation coefficient calculation unit 224, a coordinate value estimation unit (transform estimation unit) 226, and The coordinate values of undefined vertices are estimated and output from the triangular patches near the area.

The concealment area estimation unit 44 estimates the pixel value of the concealment area from the triangular patch estimated from the vertex position estimation unit 42 and the effective area of each object from the object detection unit 40 and the image of the input frame. Is. The difference of this circuit from the above embodiment is that in the above embodiment, each triangular patch is normalized before the pixel value of the concealed area is obtained. However, in the present embodiment, since the shape region calculation unit 46 is provided in the subsequent stage, the same configuration as the texture estimation unit 26 of the second embodiment described above may be used. For example, as shown in FIG. 18, a reference frame detection unit 260, an affine transformation coefficient calculation unit 262, a reference pixel position calculation unit 264, and a texture estimation unit (pixel value estimation unit) 266 are included and reference frame triangular patches are included. And the triangular patch of the hidden area are associated with each other by shape conversion, and the pixel value of the hidden area is estimated by their linear prediction.

The shape area calculation unit 46 is a hidden area estimation unit.
The contour data is generated based on each data interpolated in 44, that is, each object is divided into triangular patches in the same manner as in the first embodiment not including the concealed area, and each triangular patch is normalized. Then, the contour data represented by the triangular patch and the contour distance is generated.

The interpolation object image generation unit 48 is an image processing device that generates an interpolation image of each object based on the contour data from the shape area calculation unit 46, and in the present embodiment, as shown in FIG. The configuration is such that the second affine transformation unit 308 of the second embodiment is omitted.

In the operating state, first, when the frame at time t0 and each layer image are supplied, the object detection unit 40 detects the effective area of each object, and the result is sent to the hidden area estimation unit 44. Supply.
At this time, the vertex position estimation unit 42 receives the data of the vertex file and detects whether or not each object has a hidden area. For example, when the object has no hidden area in the frame at time t0, the data of each input image and the data representing the effective area of the object are supplied to the shape area calculation unit 46 via the hidden area estimation unit 44. As a result, the shape region calculation unit 46 forms the contour data of each object as in the first embodiment, and supplies the result to the interpolation object image generation unit 48.

Next, for example, when the image of the frame at time t10 shown in FIG. 20 and the layer images shown in FIGS. 21 to 23 are supplied to the object detecting section 40, the object detecting section 40 displays, for example, in FIG. Data of the effective area of the indicated object is generated, and the result is supplied to the hidden area estimation unit 44. At this time, the vertex position estimating unit 42 detects the hidden area from the data of the vertex file and the layer image, detects the vertex position of the area, and outputs the result to the hidden area estimating unit.
Supply to 44. Accordingly, the hidden area estimation unit 44 estimates the pixel value of the hidden area from the reference patch of the frame at time t0, and supplies the result to the shape area calculation unit 46.

The shape region calculation unit 46 generates contour data of the object at time t10 as in the above, and supplies the result to the interpolation object image generation unit 48. As a result, the interpolation object image generation unit 48 causes the time t0 and the time t1.
For example, the respective object in the interpolation frame at time t5 is generated from the respective contour data of 0, and the generated object image is supplied to the interpolation image synthesizing unit 14.

As a result, the interpolation image synthesizing unit 14 sequentially maps the object images spatially rearward into one frame to form an interpolation frame, and outputs the result.

As described above, according to the interpolation image generation apparatus of this embodiment, the effective area of an object is detected from the input frame and each layer image, and if the object has a hidden area, the pixels of the hidden area are detected. Is interpolated, and contour data is generated based on the interpolated object. Therefore, an interpolated image can be effectively generated even from an object including a hidden area. In this case, by generating the contour data, it is possible to obtain an image with less distortion of the contour shape. Further, the circuit for interpolating the hidden area can be simplified, and the amount of calculation can be further reduced.

Next, FIG. 25 shows a fourth embodiment of the interpolation image generating apparatus according to this embodiment. The interpolation image generation apparatus according to the present embodiment is, for example, an image processing apparatus that inputs a cell image of each object such as animation and generates an interpolation frame thereof, and as shown in the figure, an object area detection unit 40 and A shape area calculation unit 46, an interpolation object image generation unit 48, and an interpolation image synthesis unit 14 are included.

The object area detection unit 40 detects the effective area of each object from the cell image and the layer image at each time, and supplies the result to the shape area calculation unit 46.

The shape area calculation unit 46 receives the output from the object area detection unit 40 and the vertex data from the object data file, divides each object into triangular patches, and generates contour data in the same manner as above. . The result is supplied to the interpolation object image generation unit 48.

Similarly to the above, the interpolation object image generating unit 48 generates the objects of the interpolation frames from the adjacent frames and supplies the results to the interpolation image synthesizing unit 14.

The interpolated image synthesizing unit 14 sequentially maps the object images from the interpolated object image generating unit 48 from one spatially rearward to one frame to form an interpolated frame, and outputs the result. .

As described above, according to the interpolation image generation apparatus of this embodiment, an image for each object having no hidden area such as animation is input, an interpolation image is generated for each object, and they are combined. Thus, the interpolation frame can be effectively generated. Further, by processing the input image for each object and generating the interpolated image for each object, it is possible to generate the interpolated image from the moving image in which a plurality of objects having different movements exist. Further, since the input object does not have a hidden area, the circuit for estimating the hidden area can be omitted, and the processing can be executed with a simple circuit configuration.
For example, in animation, there is a method in which an image for each object is created for each frame, the positions of the objects are slightly shifted, and images are taken for each frame to create a moving image. However, many steps are required for completion. In the present embodiment, one cell image for each object is formed in several frames, and the positions of the triangular patches for dividing the object are designated, so that a frame can be automatically created without the creation process. As a result, when an artificial moving image such as an animation is generated, it is possible to reduce the number of generation steps.

In each of the above embodiments, the triangular patch obtained by dividing the object is processed by normalizing it with an isosceles right triangle, but in the present invention, the triangular patch is processed with another triangle such as an equilateral triangle. It may be normalized. Further, in each of the above embodiments, the object is divided into triangular patches for processing, but other polygons such as a quadrangle may be used in the present invention. In this case, when the patch is a quadrangle, it is preferable to use co-linear transformation or perspective transformation instead of affine transformation.

[0152]

As described above, according to the present invention, when representing the contour shape of an object in a moving image, the object is divided into polygonal patches, and the patches are normalized with a predetermined polygonal shape. Since the distance between the side of the patch and the contour is obtained and the contour shape is represented by the contour distance and the patch, the image can be effectively processed in the area along the contour of the object. In this case, it is possible to effectively generate the interpolated image without increasing the amount of calculation and without distorting the contour region.

Further, according to the present invention, when an object in a moving image has a region hidden by another object, the object is divided into polygonal patches to detect a patch in the hidden region, and the patch of the patch is detected. Since the vertex position and contour shape and the pixel value of the hidden area can be predicted by referring to the patches of other frames and the hidden area can be effectively interpolated, the object image is used even when the object has the hidden area. It is possible to effectively form an interpolated image for each object.

[Brief description of drawings]

FIG. 1 is a functional block diagram showing a first embodiment of an interpolation image generation apparatus to which a contour data generation method according to the present invention is applied.

FIG. 2 is a functional block diagram showing a configuration example of a shape calculation unit of the interpolation image generation device according to the embodiment of FIG.

FIG. 3 is a functional block diagram showing a configuration example of an interpolation object image generation unit of the interpolation image generation device according to the embodiment of FIG.

FIG. 4 is a frame diagram showing an example of object division applied to the interpolation image generation apparatus according to the embodiment of FIG.

5 is a diagram showing an example of normalization of triangular patches in the interpolation image generating apparatus according to the embodiment of FIG.

FIG. 6 is a diagram showing an example of estimation of vertex positions in a triangular patch in the interpolation image generation device according to the embodiment of FIG.

FIG. 7 is a diagram showing an example of estimation of triangular patches of an interpolation image in the interpolation image generation device according to the embodiment of FIG.

FIG. 8 is a diagram illustrating an example of estimating pixel values of an interpolated image in the interpolated image generation device according to the embodiment of FIG.

FIG. 9 is a diagram showing an example of generation of an interpolation frame in the interpolation image generation device according to the embodiment of FIG.

FIG. 10 is a functional block diagram showing a second embodiment of the interpolation image generation apparatus to which the hidden area estimation method according to the present invention is applied.

11 is a functional block diagram showing a configuration example of a vertex coordinate value estimation unit of the interpolation image generation device according to the embodiment of FIG.

12 is a functional block diagram showing a configuration example of a hidden area estimation unit of the interpolation image generation device according to the embodiment of FIG.

13 is a functional block diagram showing a configuration example of a texture estimation unit of the interpolation image generation device according to the embodiment of FIG.

14 is a functional block diagram showing a configuration example of an interpolation object image generation unit of the interpolation image generation device according to the embodiment of FIG.

FIG. 15 is a diagram for explaining the operation of the interpolation image generating apparatus and the hidden area estimation method according to the embodiment of FIG.

FIG. 16 is a functional block diagram showing a third embodiment of the interpolation image generation device according to the present invention.

17 is a functional block diagram showing a configuration example of a vertex position estimation unit of the interpolation image generation device according to the embodiment of FIG.

18 is a functional block diagram showing a configuration example of a hidden area estimation unit of the interpolation image generation apparatus according to the embodiment of FIG.

19 is a functional block diagram showing a configuration example of a vertex position estimation unit of the interpolation image generation device according to the embodiment of FIG.

20 is a diagram showing an example of an input image to the interpolation image generation device according to the embodiment of FIG.

FIG. 21 is a diagram showing an example of a layer image to the interpolation image generation device according to the embodiment of FIG.

22 is a diagram showing an example of a layer image to the interpolation image generation device according to the embodiment of FIG.

FIG. 23 is a diagram showing an example of a layer image to the interpolation image generation device according to the embodiment of FIG. 16.

FIG. 24 is a diagram showing an effective area of an object in the interpolation image generating apparatus according to the embodiment of FIG.

FIG. 25 is a functional block diagram showing a fourth embodiment of the interpolation image generation device according to the present invention.

[Explanation of symbols]

10,46 Shape calculator 12,30,48 Interpolation object image generator 14 Interpolation image composition section 20 Hidden area interpolator 22,42 Vertex coordinate value estimation unit 24,44 Concealed area estimation unit 26 Texture Estimator 40 Object detector 100 Affine transform coefficient calculator 102 Affine transformation unit 104 Contour distance calculator

Front page continuation (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06T ⁷ /00-7/60 G06T 9 / 20,13 / 00 H04N 7/24-7/50

Claims (10)

(57) [Claims] 1. An interpolation image generation apparatus for respectively forming an interpolation frame between a plurality of frames representing a moving image, wherein the apparatus detects contour data representing contour shapes of an object of each frame. A detection unit, an object image generation unit that estimates an interpolated image of an object based on the contour data from the contour data detection unit and generates the image, and an interpolated frame that combines the object image from the object image generation unit The contour data detecting unit divides the object into polygonal regions, and transforms the shape deformation between the divided patches and a predetermined polygon serving as a reference. A conversion coefficient calculating means for obtaining the coefficient and that of the area inside the divided patch and between the patch and the contour of the object Normalizing means for converting and normalizing the coordinate values by using the conversion coefficient obtained by the conversion coefficient calculating means, and the distance from the side of the patch converted by the normalizing means to the contour of the object vertically And a contour data forming means for forming contour data representing the shape of the contour by using the value obtained by the contour distance detecting means and the coordinate values of the vertices of the patch. Interpolation image generator. 2. The apparatus according to claim 1, wherein the conversion coefficient calculation means divides the object into patches of arbitrary triangles, and affines between the triangular patches and a reference isosceles right triangle. An interpolation image generation device characterized by obtaining a conversion coefficient. 3. The apparatus according to claim 2, wherein the normalizing means means uses the affine transformation coefficient obtained by the transformation coefficient calculating means for each of the regions between the triangular patch and the contour of the object. An interpolation image generation device characterized by normalizing the coordinates of an object by affine transforming the coordinate values of points. 4. The apparatus according to claim 1, wherein the object image generation unit estimates the coordinate value of the vertex of the patch of the object of the interpolation image from the coordinate value of the vertex of the patch of the object of the adjacent frame. An estimating means; a first transform coefficient calculating means for obtaining a transform coefficient of the shape deformation of the patch of the interpolation image from the patch of the interpolation image estimated by the vertex position estimating means and a predetermined polygon serving as a reference; Contour distance estimating means for estimating the distance between the patch and the contour in the interpolated image based on the distance between each patch and the contour in the adjacent frame from the data detecting means, and the vertex position estimating means for estimating The contour data represented by the patch of the interpolated image and the contour distance estimated by the contour distance detecting means is used as the conversion coefficient obtained by the conversion coefficient calculating means. A second transform coefficient for transforming the patch and contour of the interpolation image by transforming, and a transform coefficient of the shape deformation from the patch of the adjacent frame and the patch of the interpolated image estimated by the vertex position estimating means. The calculation unit and the adjacent frame using the conversion coefficient obtained by the second conversion coefficient calculation unit are associated with each pixel of the interpolation image whose shape is converted by the conversion unit, and the interpolation is performed based on the result. An interpolated image generation apparatus, comprising: a texture estimation unit that estimates a pixel value of an image from pixel values of adjacent frames to estimate a texture of an interpolated image. 5. The apparatus according to claim 4, wherein each object is divided into triangular patches, and the first conversion coefficient calculation means is a triangle of the interpolated image estimated by the vertex position estimation means. An affine transformation coefficient is obtained between the patch and the reference isosceles right triangle, and the second transformation coefficient calculation means obtains the affine transformation coefficient from between the triangular patches in the image of the adjacent frame and the interpolation image. An interpolated image generation device characterized by: 6. The apparatus according to claim 5, wherein the transforming unit transforms the shape of the triangular patch and the shape region of the interpolated image using the affine transform coefficient calculated by the first transform coefficient calculating unit, The interpolation image generating apparatus, wherein the texture estimating means converts and associates the triangular patch of the adjacent frame and the triangular patch of the interpolating frame with the affine transform coefficient obtained by the second transform coefficient calculating means. 7. An object of each frame of a moving image
Patch transfected Te outline data generation method odor <br/> for detecting a contour of, the method includes dividing the object in a polygonal area, divided
Transformation coefficient of the shape deformation between the specified polygon and the reference
The first step of finding the transformation coefficient between the edges of the patch and the contour of the object.
The second step of converting using a number, and the vertical conversion from an arbitrary position on the side of the converted patch.
And a third step of calculating a distance to a contour of the patch from the edge of the patch and the coordinate value of the vertex of the patch.
A contour data generation method characterized in that a contour shape is expressed by using a distance to the contour . 8. The method according to claim 7, wherein the first step divides the object into triangular regions.
Then, the divided triangular patch and the reference right-angled isosceles triangle
And the affine transformation coefficient obtained in the first step.
Each point of the triangular patch and the contour and
Contour data characterized by affine transformation of each point between
Data generation method . 9. The contour data generating method according to claim 7.
Image of the interpolated frame based on the contour data formed by
A method for generating an interpolated image, the method comprising : coordinates of vertices of patches of objects in adjacent frames.
From the value, the coordinate value of the vertex of the patch of the object of the interpolation image
Of the interpolated image estimated in the first step and a predetermined reference
Transformation coefficient of the patch shape deformation of the interpolation image from the polygon of the
Of the second step of obtaining the contour data and the contour data generated by the contour data generation method.
Each patch in the frame adjacent to the interpolated frame
And the contour in the interpolated image based on the distance between
A third step of estimating the distance between the interpolated image and the patch of the interpolated image estimated in the first step, and
The ring represented by the contour distance estimated in the third step
Convert the contour data with the conversion coefficient obtained in the second step.
The shape of the interpolated image patch and contour
Process, the patch of the adjacent frame, and the supplement estimated in the first process.
Calculate the transformation coefficient of shape deformation from the patch of the inter-image
5 and an adjacent frame using the conversion coefficient obtained in the step
Each of the interpolated images whose shape has been converted in the fourth step is
Corresponds to the pixel and the pixel of the interpolated image based on the result
The value of the interpolated image
Interpolation image generation method characterized by comprising a sixth step of estimating the texture. 10. The method for generating an interpolated image according to claim 9 , wherein the method includes the steps from the first step to the sixth step.
And generate an interpolated image for each object.
Interpolation image generation method comprising that you generate an interpolation frame Luo object image synthesis to.