JP5025432B2 - Frame interpolation apparatus and frame interpolation method - Google Patents

Description

  The present invention relates to a frame interpolation apparatus and a frame interpolation method for generating an image of an interpolated frame existing between two frames using two frames of images included in a video signal.

  A hold-type display such as a liquid crystal display continues to display the same image for one frame period, and when the object in the image moves, the tracking of the human eye to the moving object moves continuously, There is a problem that the edge portion looks blurry because the object moves discontinuously in units of one frame. On the other hand, it is conceivable to increase the number of display frames by interpolating frames to make the object move smoothly.

  In addition, for materials in which film images such as movies are converted into television signals, two (2 fields) or 3 (fields) are created from the same frame because of the difference in frame frequency between the two (film images and television signals). If the image is displayed as it is, there is a problem that the motion is blurred or judder with a jerky motion occurs.

  Similarly, a material obtained by converting computer-processed video into a TV signal is an image in which two (2 fields) are made from the same frame.

  As a conventional frame interpolation apparatus and frame interpolation method, a zero hold method in which interpolation is performed with the same image as the previous frame with respect to the interpolation frame, or an image before and after the first frame with respect to the interpolation frame. Although there is an average value interpolation method for interpolating with an average image, the zero hold method does not move smoothly with respect to an image moving in a fixed direction, so the problem of blurring of the hold type display is still not solved. Further, the average value interpolation method has a problem that a moving image becomes a double image.

  As an improvement measure, an image is divided into a plurality of pixel blocks, a frame one frame before the interpolation frame is used as a first reference frame, and a frame one frame after the interpolation frame is used as a second reference frame. Find a pixel block having the highest correlation with the pixel block in the first reference frame from the second reference frame, and interpolate an image of the pixel block of the interpolation frame from the found pixel block pair (for example, Patent Reference 1). In this method, since an interpolation frame is generated in units of pixel blocks, data read / write in a memory (mainly SDRAM is used) necessary for processing becomes complicated, and hardware implementation is difficult. There is.

  Further, as another improvement measure, from the pixel having the highest correlation between the pixel on the first reference frame and the pixel on the second reference frame that is point-symmetric with respect to the interpolation pixel of the interpolation frame. Some generate an interpolation pixel of an interpolation frame (see, for example, Patent Document 2). In this method, because correlation is detected in units of pixels, it may be detected that the correlation between pixels is large even though the contents of the image are different, and an interpolated image may not be obtained correctly.

JP 2004-357215 A (page 9, FIG. 16) Japanese Patent Laying-Open No. 2006-129181 (page 8, FIG. 3)

  The conventional frame interpolating apparatus is configured as described above, and there is a problem that motion is blurred or judder in which motion is jerky occurs. In addition, there is a problem that the correlation cannot be detected correctly in the method of detecting the correlation in pixel units.

The frame interpolation apparatus of the present invention
A frame interpolation device for generating a pixel value of an interpolation pixel in an interpolation frame between a first reference frame and a second reference frame,
One of the pixel block pair consisting of the pixel block of the first reference frame and the pixel block of the second reference frame that is point-symmetric with respect to the position of the interpolation pixel, and the pixel block pair Block pair extracting means for extracting a plurality of one or both of the pixel block pairs in which the pixel blocks are replaced with pixel blocks shifted by one pixel;
Similarity calculation means for calculating the similarity of two pixel blocks based on the pixel values of the pixels in the two pixel blocks for each of a plurality of pixel block pairs extracted by the block pair extraction means;
The plurality of block pairs extracted by the block pair extraction unit are divided into a plurality of areas such that those in which the directions of straight lines connecting the central pixels of the pixel blocks of each block pair are similar belong to the same area. Area dividing means to divide into
In each area divided by the area dividing means, an area representative extracting means for selecting a pixel block pair having the highest similarity calculated by the similarity calculating means as an area representative block pair;
Among the plurality of area representative block pairs extracted by the area representative extracting unit, a pixel block pair having the largest similarity calculated by the similarity calculating unit or a corrected one is selected as a candidate block pair. In addition, among the plurality of area representative block pairs extracted by the area representative extracting means, the similarity calculated by the similarity calculating means or a correction of the same among the other than the candidate block pairs Candidate block pair extraction means for selecting the pixel block pair having the largest as the evaluation block pair;
A reliability evaluation means for evaluating the reliability of the candidate block pair based on the similarity of the candidate block pair or the corrected version thereof and the similarity of the evaluation block pair or the size of the corrected block pair; and
Reference pixel position selection means for selecting a center position of each pixel block constituting the candidate block pair as a reference pixel position;
Symmetric position interpolation means for generating the interpolation pixel based on the pixel value of the reference pixel position selected by the reference pixel position selection means,
The reference pixel position selection means, when the reliability evaluation means determines that there is reliability, selects the center position of each pixel block constituting the candidate block pair as a reference pixel position, and has no reliability. If it is determined, the reference pixel position selected before is selected as the reference pixel position of the interpolation pixel.

  According to the present invention, each of a pair of pixel blocks having a relative positional relationship (direction and distance) closest to the amount of image motion between the first reference frame and the second reference frame having the highest similarity. Since the pixel value of the interpolation pixel is determined based on the pixel value of the pixel located at the center of the pixel, correct correlation detection can be performed. Therefore, accurate interpolation can be performed, and motion blur and judder are improved. Also, a representative block pair for each area in which block pairs having a close relative position relationship are collected is obtained, and among them, the block having the highest similarity is set as a candidate block pair, and the block having the next highest similarity is evaluated. Since the reliability is evaluated as described above, it is possible to accurately determine the reliability.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
Embodiment 1 FIG.
FIG. 1 is a block diagram showing a frame interpolation apparatus according to Embodiment 1 of the present invention. The illustrated frame interpolation apparatus includes an input terminal 1, a delay unit 2, a block pair extraction unit 3, a similarity calculation unit 4, an area division unit 5, an area representative extraction unit 6, a candidate extraction unit 7, Reliability evaluation means 8, reference pixel position selection means 9, delay means 10, symmetrical position interpolation means 11, and output terminal 12 are provided.

The video signal input from the input terminal 1 is input to the delay unit 2, delayed by one frame period, and input to one of the block pair extraction units 3 as a first reference frame. The video signal input to the input terminal 1 is input to the other input of the block pair extraction unit 3 as a second reference frame. The block pair extracting means 3 is located substantially symmetrical with respect to the interpolated pixel from the first reference frame and the second reference frame, and extracts a plurality of pairs of pixel blocks composed of the same number of pixels. Output to the calculation means 4.
The block pair extraction unit 3 also outputs the pixel value at the center position of each pixel block to the symmetric position interpolation unit 11.

  As shown in FIG. 2, the pixels in each frame are arranged in a matrix in the frame F (on the screen constituting the frame). That is, they are aligned with each other in the horizontal direction (main scanning direction) and the vertical direction (sub-scanning direction). The position of each pixel in the frame F is represented by (h, v) by a horizontal coordinate value h and a vertical coordinate value v. The coordinate origin is at the leftmost and uppermost position of the frame, the horizontal coordinate value h increases rightward by one for each pixel interval, and the vertical coordinate value v decreases downward. Assume that the value increases by 1 for each interval.

  When generating an interpolation pixel at an arbitrary position (h, v) in the interpolation frame, as the pixel block BLa in the first reference frame and the pixel block BLb in the second reference frame, for example, the center position thereof is in the horizontal direction. Are extracted in the range from (h-2) to (h + 2) and in the vertical direction are extracted from (v-1) to (v + 1). There are 15 pixel blocks satisfying such conditions in each of the first reference frame and the second reference frame. Each pixel block is composed of, for example, pixels of 3 rows and 5 columns.

  3A to 3D and FIGS. 4A to 4D illustrate some of such pixel blocks. Among them, what are shown in FIGS. 3A to 3D are pixel blocks BLa in the first reference frame, and those shown in FIGS. 4A to 4D are those in the second reference frame. This is a pixel block BLb.

  The pixel block BLa (1) shown in FIG. 3 (a) and the pixel block BLb (1) shown in FIG. 4 (a) constitute a pixel block pair composed of pixel blocks located at point symmetry with respect to the interpolation pixel. is doing. Similarly, a pixel block consisting of a pixel block in which the pixel block BLa (2) shown in FIG. 3B and the pixel block BLb (2) shown in FIG. Make up a pair. Similarly, a pixel block consisting of a pixel block in which the pixel block BLa (3) shown in FIG. 3C and the pixel block BLb (3) shown in FIG. 4C are symmetric with respect to the interpolation pixel. Make up a pair. Similarly, a pixel block composed of pixel blocks BLa (15) shown in FIG. 3D and pixel blocks BLb (15) shown in FIG. Make up a pair.

  In this way, the block pair extraction unit 3 sequentially extracts pixel block pairs composed of two pixel blocks located at point-symmetric positions with the interpolation pixel as the center.

  The block pair extraction unit 3 further includes not only a pair of pixel blocks at a point-symmetrical position with respect to the interpolation pixel as shown in FIGS. 3A to 3D, but also one of them as one pixel. Also extracted are those replaced with pixel blocks shifted to the left. For example, the pixel block BLb (1) shown in FIG. 4A is shifted to the left by one pixel as shown in FIG. 5A, and the pixel block BLb (1) shown in FIG. A pixel block pair consisting of a block BLa (1) and a pixel block BLb in which the pixel block BLb (2) shown in FIG. 4B is shifted to the left by one pixel as shown in FIG. 5B. FIG. 5C shows a pixel block pair composed of (2) and the pixel block BLa (2) shown in FIG. 3B. Similarly, the pixel block BLb (3) shown in FIG. As shown in FIG. 4, a pixel block pair composed of the pixel block BLb (3) shifted to the left by one pixel and the pixel block BLa (3) shown in FIG. The pixel block BLb (15) shown is shown in FIG. A pixel block BLb (15) which is shifted by one pixel left so that, to extract the composed pixel block pair out with pixel block BLa (15) shown in Figure 3 (d).

  The pixel blocks illustrated in FIGS. 3A to 3D and the pixel blocks illustrated in FIGS. 4A to 4D are horizontally aligned with each other (between the pixels corresponding to each other inside them). There is a difference (distance) in the position of an even number of pixels, that is, 4 pixels, 2 pixels, and 0 pixels, but the pixel blocks shown in FIGS. 3A to 3D and FIGS. The pixel blocks shown in (1) have an odd number of pixels in the horizontal direction, that is, a difference (distance) of 3 pixels and 1 pixel.

  The block pair extraction means 3 further constitutes a pair of pixel blocks located at point-symmetric positions with the interpolation pixel as the center, as illustrated in FIGS. 3 (a) to 3 (d) and FIGS. 4 (a) to 4 (d). A pixel block in which one of the pixel blocks to be replaced is replaced with a pixel block shifted upward by one pixel is also extracted. For example, a pixel block constituted by a pixel block obtained by shifting the pixel block illustrated in FIGS. 4A to 4D by one pixel and a pixel block illustrated in FIGS. 3A to 3D. Extract pairs.

  The block pair extraction means 3 further constitutes a pair of pixel blocks located at point-symmetric positions with the interpolation pixel as the center, as illustrated in FIGS. 3 (a) to 3 (d) and FIGS. 4 (a) to 4 (d). A pixel block in which one of the pixel blocks to be replaced is replaced with a pixel block shifted to the left by one pixel and upward by one pixel (that is, obliquely to the upper left) is also extracted. For example, a pixel constituted by a pixel block in which the pixel block illustrated in FIGS. 4A to 4D is shifted one pixel diagonally to the upper left and a pixel block illustrated in FIGS. 3A to 3D. Extract block pairs.

  The pixel block pair extracted by the block pair extraction unit 3 is input to the similarity calculation unit 4 and the similarity between the pixel blocks is calculated for each pixel block pair. For example, when calculating the similarity between the pixel block BLa in FIG. 3A and the pixel block BLb in FIG. 4A, the similarity calculation unit 4 calculates the pixels a1 to 1 in the pixel block BLa in FIG. The sum of absolute differences (hereinafter referred to as “a15”) and the pixels b1 to b15 (pixels at the positions corresponding to the pixels a1 to a15 (the same positions in the pixel block)) in the pixel block BLb of FIG. (Referred to as SAD) is calculated as the similarity according to the following equation.

Similarly, the sum SAD of absolute difference values is calculated for other pixel block pairs, and the calculation result is output to the area representative extracting means 6 as the similarity of each pixel block pair. Here, the smaller the sum SAD value, the greater the similarity, and the greater the sum SAD, the smaller the similarity.
In this way, the sum SAD of absolute difference values (index indicating similarity) is calculated for all the block pairs extracted by the block pair extraction means 3. In this example, since 15 block pairs are extracted for each of the above four types, the sum SAD of absolute difference values is calculated for a total of 60 block pairs.

  The 60 block pairs extracted by the block pair extracting means 3 are different from each other in the direction and length of the straight line connecting the central pixels of the pixel blocks of each block pair (distance between the central pixels). The relative positional relationship between the block pairs can be defined by the direction and length of each block. As will be described later, when the degree of similarity between the pixel blocks of the block pair is the highest, it corresponds to the direction of the straight line connecting the central pixels of those blocks and the direction and size of the length movement. Therefore, the direction of the straight line connecting the central pixels of the block may be referred to as the “movement direction”.

FIG. 6 shows an example of the matrix arrangement of the relative positional relationship described above.
In this example (examined example), since the block pair extraction unit 3 extracts 60 block pairs, the matrix in FIG. 6 has 60 element positions arranged in 6 rows and 10 columns. These 60 element positions respectively correspond to the relative positional relationship between the block pairs, and therefore it can be said that each element corresponds to the block pair.
The position of each element in the matrix M is represented by (i, j) by a horizontal coordinate value (column number) i and a vertical coordinate value (row number) j. The coordinate origin is at the leftmost and uppermost position of the matrix, the horizontal coordinate value i increases to the right by one for each element interval, the vertical coordinate value j decreases downward, Assume that the value increases by 1 for each interval.

The element at the leftmost and uppermost position (i = 1, j = 1) of the matrix is composed of the pixel block shown in FIG. 3 (a) and the pixel block shown in FIG. 4 (a). Corresponds to a block pair.
The element at the position of i = 3 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3B and the pixel block shown in FIG.
The element at the position of i = 5 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3C and the pixel block shown in FIG.
The element at the position of i = 9 and j = 5 corresponds to a block pair composed of the pixel block shown in FIG. 3D and the pixel block shown in FIG.
Generally speaking, when f = 0, 1, 2, 3 or 4, g = 0, 1 or 2, the position of i = 1 + (2f), j = 1 + (2g) (in FIG. The elements in (indicated by “”) are point-symmetrical with respect to the interpolated pixel as illustrated in FIGS. 3A to 3D and FIGS. 4A to 4D. This corresponds to a pixel block pair composed of pixel blocks.

The element at the position of i = 2 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3A and the pixel block shown in FIG.
The element at the position of i = 4 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3B and the pixel block shown in FIG.
The element at the position of i = 6 and j = 1 corresponds to a block pair composed of the pixel block shown in FIG. 3C and the pixel block shown in FIG.
The element at the position of i = 10 and j = 5 corresponds to a block pair composed of the pixel block shown in FIG. 3D and the pixel block shown in FIG.

  Generally speaking, when f = 0, 1, 2, 3 or 4, g = 0, 1 or 2, the position of i = 2 + (2f), j = 1 + (2g) (“Δ” in FIG. As shown in FIGS. 3A to 3D and FIGS. 5A to 5D, the element in FIG. 3A is a pixel block in a point-symmetric position with respect to the interpolation pixel. Corresponds to a pixel block pair in which the pixel block of the second reference frame is replaced with a pixel block shifted to the left by one pixel.

Similarly, when f = 0, 1, 2, 3 or 4, g = 0, 1 or 2, the position of i = 2 + (2f), j = 1 + (2g) (indicated by “□” in FIG. 6). The element in (shown) is replaced with a pixel block in which the pixel block of the second reference frame is shifted up by one pixel with respect to a pixel block pair consisting of pixel blocks located in point symmetry with respect to the interpolation pixel. Pixel block pair (a pixel block composed of a pixel block obtained by shifting the pixel block shown in FIGS. 4A to 4D by one pixel and a pixel block shown in FIGS. 3A to 3D) Corresponding to).
Similarly, when f = 0, 1, 2, 3 or 4, g = 0, 1 or 2, the position of i = 2 + (2f), j = 2 + (2g) (indicated by “x” in FIG. 6). The element in (shown) is replaced with a pixel block in which the pixel block of the second reference frame is shifted one pixel diagonally to the upper left with respect to a pixel block pair consisting of pixel blocks located symmetrically with respect to the interpolation pixel. The pixel block pair (the pixel block shown in FIGS. 4A to 4D is shifted by one pixel diagonally to the upper left and the pixel block shown in FIGS. 3A to 3D). Corresponds to a pixel block pair).

The position of each element shown in FIG. 6 corresponds to the relative positional relationship of the block pair consisting of the block of the first reference frame and the block of the second reference frame as described above. This corresponds to the direction and magnitude of the motion vector between the block pairs.
FIG. 7 schematically shows the direction and magnitude of the motion vector at each of the 60 element positions in FIG. 6 by arrows. The direction and length of the arrow in FIG. 7 corresponds to the direction and magnitude of the motion vector.

For example, the element position of i = 1 and j = 1 is a large downward diagonal movement from the block of FIG. 3A of the first reference frame to the block of FIG. 4A of the second reference frame. Corresponding, the arrow at that position is long and slanted to the lower right.
The element position of i = 5 and j = 1 corresponds to the downward movement from the block of FIG. 3 (c) of the first reference frame to the block of FIG. 4 (c) of the second reference frame. The arrow at that position is pointing downward.

The area dividing means 5 divides according to the plurality of element positions shown in FIG. 6, that is, the relative positional relationship of the block pairs, or the directions of the plurality of motion vectors shown in FIG. Try to belong to a group. Therefore, it can be said that the area dividing unit 5 is a classifying unit that classifies block pairs according to the direction of the motion vector (classifies block pairs whose motion vectors are close to each other to belong to the same group).
For example, as shown in FIG. 7, 60 vectors (and thus the block pairs of FIG. 6) are divided into nine areas A to I as shown by bold lines in FIG. 7 and further as shown in FIG. Divide into

  For example, i = 1, j = 1 block pair, i = 2, j = 1 block pair, i = 3, j = 1 block pair, i = 1, j = 2 block pair, i = 2, j = 2 block pair, i = 3, j = 2 block pair, i = 1, j = 3 block pair, i = 2, j = 3 block pair, i = 3, j = 3 block pair Be belonging to area A. This area A is a vector area that is diagonally downward to the right.

  When area B is divided similarly to area I, area B is a downward vector area, area C is a diagonally downward vector area, area D is a right vector area, area E is a static vector area without motion, area F Is a leftward vector area, area G is a diagonally upward vector area, area H is an upward vector area, and area I is a diagonally upward vector area.

The area representative extracting means 6 has the largest similarity calculated by the similarity calculating means 4 among the plurality of block pairs belonging to each of the areas A to I by the area dividing means 5 (the sum of absolute differences). A pixel block pair having the smallest SAD is selected and extracted as an area representative block pair.
One area representative block pair is extracted from each of area A to area I, and a total of nine area representative block pairs are extracted.

The candidate extracting means 7 selects a pixel block pair (area representative block pair) having the highest similarity calculated by the similarity calculating means 4 among the area representative block pairs of each area extracted by the area representative extracting means 6. Select as a candidate block pair.
The candidate extracting unit 7 also has a pixel block having the highest similarity among the plurality of area representative block pairs extracted by the area representative extracting unit 6 other than the candidate block pair (area representative block pair). A pair (a pixel block pair with the second highest similarity calculated by the similarity calculation unit 4 among the area representative block pairs of each area extracted by the area representative extraction unit 6) is selected as an evaluation block pair. To do.

The reliability evaluation means 8 calculates the average value of the similarity of the nine area representative block pairs extracted by the area representative extraction means 6 (average value of the sum SAD of the absolute differences), and evaluates the average value of the similarity and the evaluation. The reliability of the candidate similarity is evaluated from the similarity of the block pair (evaluation similarity) and the similarity of the candidate block pair (candidate similarity). For example, if the following inequality (2) holds, it is determined that there is reliability, and if it does not hold, it is determined that there is no reliability.
(Se−Sc) × B + C> (Sa−Se) (2)
(Se is a value indicating similarity for evaluation (total sum SAD of difference absolute values of evaluation block pair),
Sc is a value indicating the candidate similarity (the sum SAD of absolute differences of candidate block pairs),
Sa is a value indicating the average value of the similarity of the area representative block pair (the average value of the sum SAD of the absolute difference values of the area representative block pair),
B is a coefficient greater than 1, for example 2,
C is a constant greater than or equal to 0, for example, 100 (assuming that the pixel value is represented by 8 bits, and therefore the maximum possible value is 255).

  The reliability evaluation unit 8 outputs the result of determination according to the above equation (2) to the reference pixel position selection unit 8.

  The expression (2) is satisfied when the value obtained by adding C to B times the difference between the candidate similarity and the evaluation similarity (B> 1) is the evaluation similarity (for all area representative block pairs). This means that it is larger than the difference from the average value of the similarity, that is, the similarity of the candidate block pair is remarkably large (the peak including the maximum value of the similarity is prominent).

  In the present invention, as described above, only when the similarity between the candidate block pairs is remarkably large, it is determined that the selection of the candidate block pair is reliable, and interpolation is performed using the center position of the candidate block pair as the reference pixel position. On the other hand, if the similarity is the maximum, but the difference in similarity is not so large compared to the others, the center position of the candidate block pair is not set as the reference pixel position, so that the interpolation is further performed. This is done accurately, and this can increase the accuracy of interpolation.

  Since an evaluation block pair having an area different from that of the candidate block pair is selected, even when other block pairs in the same area as the candidate block pair have a higher degree of similarity than the evaluation block pair Since the reliability evaluation is performed by eliminating such other block pairs, there are many block pairs having a higher degree of similarity than the other areas in one area (the direction of the straight line connecting the central pixels is Even when block pairs having a high degree of similarity are concentrated in the approximate range), the reliability can be evaluated appropriately.

When the reliability evaluation unit 8 determines that the reference pixel position selection unit 9 is reliable, the reference pixel position selection unit 9 selects the center pixel position of the block pair that is the output of the candidate extraction unit 7 as the reference pixel position, and the reliability is high. If it is determined that the reference pixel position has not been selected, the reference pixel position selected before, for example, the previous reference pixel position delayed by one frame by the one-frame delay means 10 is selected, and the information indicating the reference pixel position is symmetrically interpolated. Supply to means 11.
Since the selection of the reference pixel position is also the selection of the direction of the reference pixel with respect to the interpolation pixel, it also has the meaning of selecting the interpolation direction.

Here, the pixel located at the center of each pixel block (center pixel) refers to a pixel located in a central row and a central column constituting each pixel block. For example, when the pixel block is composed of pixels of 3 rows and 5 columns, it means the pixels of the second row and the third column. In general, each pixel block is composed of pixels of p rows and q columns, and when p and q are odd numbers, the pixels of the (p + 1) / 2th row and the (q + 1) / 2th column are referred to.
When p and q are even numbers, the pixel in the p / 2 row or the (p / 2) +1 row, the q / 2 column, or the (q / 2) +1 column is handled as the central pixel. That is, the center of the pixel block is not limited to the accurate center position, but may be a position near the accurate center position.
In order to uniquely determine the center position, it is desirable that p and q are odd numbers.

The symmetric position interpolation unit 11 calculates an average value of the pixel values at the reference pixel position selected by the reference pixel position selection unit 9 among the pixel values at the center position of each pixel block supplied from the block pair extraction unit 3. The calculation result is output to the output terminal 12 as the pixel value of the interpolation pixel.
By performing the above operation on all the interpolation pixels of the interpolation frame, an interpolation image of the interpolation frame is obtained.

In the above example, the block pair extraction unit 3 uses the pixel block pair (a) that is point-symmetric with respect to the interpolation pixel and one reference frame such as the second frame for such a pixel block pair. A pixel block pair (b) in which the pixel block in the reference frame is shifted to the left by one pixel (b), a pixel block pair (c) that has been shifted up by one pixel, and a pixel block pair (d) that has been shifted diagonally to the upper left by one pixel are extracted. However, the present invention is not limited to this, and in order to reduce the scale of hardware, a pixel block pair in which the pixel block in one reference frame other than the pixel block pair (a) is shifted to the left by one pixel. Only (b), only the pixel block pair (c) shifted by one pixel, or only the pixel block pair (d) shifted by one pixel diagonally to the upper left may be extracted.
Also, instead of “one pixel left”, one shifted to “one pixel right” may be used, and instead of “one pixel up”, one shifted “one pixel down” may be used, Instead of “one pixel left diagonally upper”, one shifted “one pixel right diagonally downward”, “one pixel left diagonally downward”, or “one pixel right diagonally upward” may be used.
Furthermore, a pixel block pair in which the pixel block in the first reference frame is shifted may be used instead of the pixel block pair in which the pixel block in the second reference frame is shifted.
Further, in addition to the pixel block pair composed of pixel blocks that are point-symmetric with respect to the interpolation pixel, one pixel block of the pixel block pair composed of pixel blocks that are point-symmetric with respect to the interpolation pixel is one pixel. Instead of using a pixel block pair replaced with a shifted one, a pixel block in a point-symmetrical position with respect to the interpolation pixel is used instead of a pixel block pair consisting of a pixel block in a point-symmetrical position with respect to the interpolation pixel. It is also possible to use only a pixel block pair in which one pixel block of the pixel block pair consisting of is replaced by one pixel block shifted.

  As described above, when extracting a block pair in which one pixel block of a pixel block pair composed of pixel blocks located symmetrically with respect to the interpolation pixel is shifted to one pixel, the first reference is made. Even when there is an odd number of pixels in the shifted direction between the frame and the second reference frame (the image of the subject has moved by an odd number of pixels), a pair of distances that are closest to the amount of movement The center position of each pixel block can be selected as a reference pixel position, and there is an effect that interpolation can be performed more accurately.

  For example, when extracting a block pair in which one pixel block of a pixel block pair composed of pixel blocks located symmetrically with respect to the interpolation pixel is shifted to the left by one pixel or to the right by one pixel, Even when there is an odd number of pixels in the horizontal direction between one reference frame and the second reference frame (the subject image has moved by an odd number of pixels), a pair of distances that are closest to the amount of movement The center position of each pixel block can be selected as a reference pixel position, and there is an effect that interpolation can be performed more accurately.

  Similarly, when a block pair in which one block of a pixel block pair composed of pixel blocks located symmetrically with respect to the interpolation pixel is shifted one pixel up or one pixel down is extracted, A pair having a mutual distance closest to the amount of movement even when there is an odd number of pixels in the vertical direction between the first reference frame and the second reference frame (the subject image has moved by an odd number of pixels). The center position of each pixel block can be selected as the reference pixel position, and there is an effect that interpolation can be performed more accurately.

  Similarly, one block of a pixel block pair consisting of pixel blocks that are point-symmetric with respect to the interpolation pixel is shifted one block diagonally to the upper left or one pixel diagonally to the lower right, or one pixel diagonally to the lower left. When a block pair shifted one pixel diagonally to the upper right is extracted, there is an odd pixel movement in the diagonal direction between the first reference frame and the second reference frame (the subject Even when the image is moved by an odd number of pixels), the center position of each of the pair of pixel blocks having the mutual distance closest to the amount of movement can be selected as the reference pixel position, and interpolation can be performed more accurately. There is an effect that can be done.

  On the other hand, when a pixel block pair composed of pixel blocks that are point-symmetric with respect to the interpolation pixel is extracted, there is even-numbered pixel movement between the first reference frame and the second reference frame. When the subject image is moved by an even number of pixels, the center position of each of the pair of pixel blocks having the mutual distance closest to the amount of movement can be selected as the reference pixel position, and interpolation can be performed more accurately. There is an effect that can be done.

  In addition to the pixel block pair consisting of pixel blocks that are point-symmetric with respect to the interpolation pixel, one pixel block of the pixel block pair consisting of pixel blocks that are point-symmetric with respect to the interpolation pixel is shifted to one pixel. When the block pair is extracted, even if the movement in the shifted direction between the first reference frame and the second reference frame is for odd pixels or even pixels. Even if there is even-numbered pixel movement, the center position of each of the pair of pixel blocks having the mutual distance closest to the amount of movement can be selected as the reference pixel position, and interpolation can be performed more accurately.

  When it is determined that a block pair obtained by shifting one pixel block of a pixel block pair consisting of pixel blocks located symmetrically with respect to the interpolation pixel to one pixel has the maximum similarity, Not only the position but also a position shifted by one pixel may be selected as the reference pixel position. For example, when it is determined that a block pair obtained by shifting one pixel block of a pixel block pair composed of pixel blocks located symmetrically with respect to an interpolation pixel to the left by one pixel has the maximum similarity, Not only the center position but also the pixel position adjacent to the right side of the center position is selected as the reference pixel position, the average of the pixel values of the four selected reference pixel positions is obtained, and this average is calculated as the pixel value of the interpolation pixel. It is also good to do.

  The symmetric position interpolation unit 11 uses the average value of the pixels located at the center of the pixel blocks constituting the pixel block pair as the pixel value of the interpolated pixel, but is located at the center of either one of the pixel block pairs. The pixel value of the pixel to be used may be used as the pixel value of the interpolation pixel.

  Further, as shown in FIG. 2, the similarity calculation unit 4 calculates the total SAD for all the pixels in the pixel block (from a1 to a15 and from b1 to b15), but in order to reduce the hardware scale. In addition, the sum of absolute values of the differences between the pixel values of the pixels after performing equal thinning processing on the pixels in one pixel block, for example, the pixel block BLa, and the pixels at corresponding positions in the other pixel block BLb It is good to ask for. For example, in the block BLa, pixels (r is an integer of 2 or more) are extracted in the horizontal direction, and pixels (r is an integer of 2 or more) are extracted in the vertical direction. It is also possible to extract each pixel (one for every s), obtain the absolute difference value of the pixel values of these pixels and the corresponding pixels in the block BLb, and obtain the sum thereof. An example of the thinning process when r and s are 2 is shown in FIGS. In the example shown in the figure, every other pixel in the vertical direction and the horizontal direction, and only the pixels arranged in a checkered pattern (shown by hatching) are extracted, and the absolute difference value is obtained only for these pixels, Find the sum SAD of them.

  The area dividing means 5 is divided into nine areas as shown in FIG. 7. However, the area dividing means 5 is divided into more than nine areas in order to divide the movement more finely, or nine areas are used to reduce the hardware scale. The same effect can be expected even if it is divided into smaller areas.

  As described above, in the first embodiment, a pixel block pair including the pixel block BLa of the first reference frame and the pixel block BLb of the second reference frame that is substantially point-symmetrical with respect to the position of the interpolation pixel. A plurality of extracted block pair extracting means 3 and a similarity calculating means for calculating the similarity of two pixel blocks for each of the extracted pixel block pairs based on the pixel values of the pixels in the two pixel blocks. 4 and an area dividing means 5 for dividing a detection range into a plurality of areas by using a plurality of block pairs whose movement directions are similar to the extracted pixel block pairs as one area, and within the divided areas The area representative extracting means 6 extracts the pixel block pair having the highest similarity calculated by the similarity calculating means 4 as the area representative block pair of each area. Among the area representative block pairs, the one with the highest similarity (candidate block pair) has the similarity (candidate similarity), and the area representative block pair has the next highest similarity after the candidate block pair (evaluation) Reliability evaluation means 8 for evaluating reliability based on the similarity (evaluation similarity) of the block pair), and the central pixel of the current candidate block pair according to the evaluation result by the reliability evaluation means 8 The reference pixel position selecting means 9 for selecting one of the position and the reference pixel position one frame before delayed by the delay means 10, and the pixel value of the reference pixel position selected by the reference pixel position selecting means 9 Since the symmetric position interpolating means 11 for generating the interpolation pixel is provided, the pixel block pair (D) having the largest similarity and the markedly large similarity to the other area representative block pairs. A representative block pair), and therefore, has a mutual distance closest to the amount of motion of the image between the first reference frame and the second reference frame, and the closeness of the mutual distance is another pixel block pair. Since the pixel value of the interpolation pixel can be determined based on the pixel value of the pixel located at the center of each of the pair of pixel blocks (area representative block pair) that stands out compared to it can.

  Further, a sum of absolute values of differences between pixel values of at least some of the pixels in one pixel block constituting the pixel block pair and corresponding pixels in the other pixel block is output as similarity. Therefore, there is an effect that the similarity can be obtained easily and accurately, and a more accurate interpolation image can be obtained.

  In this case, the sum of the absolute values of the differences between the pixel values of one pixel block constituting the pixel block pair and the pixel value at the corresponding position in the other pixel block is output as the similarity. For example, it is possible to obtain the similarity more accurately and to obtain a more accurate interpolated image.

  On the other hand, the sum of the absolute values of the difference between the pixel values of the pixels after the pixels of one pixel block constituting the pixel block pair and the pixels at the corresponding positions in the other pixel block are obtained. Is output as the similarity, there is an effect that the calculation of the similarity can be realized with a relatively small hardware scale.

Embodiment 2. FIG.
In the first embodiment, when the reliability evaluation unit 8 determines that there is no reliability, the previous reference pixel position delayed by one frame by the one frame delay unit 10 as the previously selected reference pixel position. However, if the reference pixel position of one frame before is erroneously interpolated, there is a possibility that the wrong position is inherited as it is and selection of an inappropriate reference pixel position is continued. In the second embodiment, the reference pixel positions around the pixel position to be interpolated are recorded, the frequency of the reference pixel positions selected from each area divided by the area dividing means 5 is analyzed, and based on the analysis result Since the reference pixel position is selected, the reference pixel position is selected more appropriately, and the generation value of the interpolated pixel is improved.

  FIG. 11 shows a block diagram of a frame interpolation apparatus according to the second embodiment. The illustrated frame interpolating apparatus is generally the same as the frame interpolating apparatus of FIG. 1, but differs in that a histogram generating means 13 is added. Components having the same reference numerals as those in FIG. 1 operate in the same manner as in the first embodiment.

  The reference pixel position selection unit 9 in FIG. 11 has the same function as the reference pixel position selection unit 9 in FIG. 1 and has additional functions as described below.

  The histogram generation means 13 obtains information representing the selected reference pixel position from the reference pixel position selection means 9, and obtains information representing the block pairs belonging to the respective divided areas from the area division means 5 to obtain reference pixels. Detecting which of the areas A to I divided by the area dividing means 5 the block pair consisting of the pixel block having the reference pixel position selected by the position selecting means 9 as the central pixel position belongs, An appearance frequency of an area to which a block pair composed of a pixel block having a reference pixel position as a central pixel position belongs is analyzed or detected, and a histogram representing the appearance frequency of each area is generated.

The range that is the detection target of the appearance frequency is a pixel that has been interpolated before the interpolation pixel (interpolation target pixel) Pa and is located in the vicinity of the interpolation pixel Pa. For example, the interpolation shown in FIG. This is a range (range surrounded by a broken line) Rh composed of 25 pixels around the pixel Pa. The surrounding 25 pixels shown in FIG. 12 are located above the interpolation pixel Pa, and among the pixels in the row whose vertical distance from the interpolation pixel Pa is equal to or less than the first predetermined value, The range from the limit position to the predetermined right limit position and the same line as the interpolation pixel Pa, located on the left side of the interpolation pixel Pa, and the range from the predetermined left limit to the left adjacent pixel of the interpolation pixel Including those in The left limit position is a position having a second predetermined value from the interpolation pixel Pa, and the right limit position is a position having a third predetermined value from the interpolation pixel Pa.
More specifically, when the coordinate value of the interpolation pixel Pa is (ha, va),
v = va-2, h = ha-5 to ha + 4,
v = va-1, h = ha-5 to ha + 4,
v = va, h = ha-5 to ha-1.
Pixels in the range of.

Then, a histogram representing the appearance frequency of the area to which the reference position pixel belongs when the pixel in the range Rh is set as an interpolation target is generated. An example of such a histogram is shown in FIG. In the example shown in FIG. 13, the area D has the highest frequency “12”, and the area E has the next highest frequency “8”.
The histogram generation unit 13 outputs information indicating the detected area having a high appearance frequency to the area representative extraction unit 6 and the reference pixel position selection unit 9.

  In the reference pixel position selection unit 9, when the reliability evaluation unit 8 evaluates that the candidate block pair of the interpolation pixel is not reliable, the area representative blocks are sequentially displayed from the area having the highest appearance frequency in the histogram generated by the histogram generation unit 13. Select a pair. For example, when the area D has the highest appearance frequency as shown in FIG. 13, the area representative extraction unit 6 performs interpolation based on the area representative block pair selected from the area D. However, if the sum SAD value of the absolute difference values of the area representative block pairs in area D is greater than or equal to the specified value T (similarity is smaller than a predetermined value), the area selected from the area E having the next highest appearance frequency Interpolate based on representative block pairs.

Embodiment 3 FIG.
FIG. 14 shows a block diagram of a frame interpolation apparatus according to the third embodiment. The illustrated frame interpolating apparatus is generally the same as the frame interpolating apparatus shown in FIG. 11, but differs in that similarity correction means 14 is added and reliability evaluation means 8 is not provided. Components having the same reference numerals as those in FIGS. 1 and 11 operate in the same manner as in the first and second embodiments.

  The frame interpolating apparatus of FIG. 14 is the same as that of the second embodiment in that the selection of the pixel block pair used for the interpolation is controlled by the histogram generated by the histogram generating means 13, but is different in the following points.

  The similarity correction unit 14 calculates the sum SAD of absolute differences indicating the similarity of the area representative block pairs of each area extracted by the area representative extraction unit 6 from the appearance frequency information of each area output from the histogram generation unit 13. Is corrected in accordance with the appearance frequency to generate a corrected similarity. For example, an area having a high appearance frequency is corrected so as to have a high degree of similarity so that it is easily determined that the reliability is high in the reliability evaluation in the next reliability evaluation means 8. For example, it is assumed that the histogram generation unit 13 sets a pixel in the range Rh shown in FIG. 12 as an interpolation target and detects a histogram as shown in FIG. 13 in the same manner as described in the second embodiment.

Assuming that the frequencies of the areas A to I are Ha to Hi and the total of the frequencies Ha to Hi (equal to the total number of pixels in the range Rh) is Ht, the similarity SAD (n) of each area (n is Any one of a to i), the calculation represented by the following formula (3) is performed to obtain the modified similarity MSAD (n).
MSAD (n) = SAD (n) × (1− (Hn / Ht) × K) (3)
(In Equation (3), K is a constant equal to or less than 1, and is predetermined, for example)
In the example shown in FIG. 13, Ht = 25.

  The smaller the value (1− (Hn / Ht) × K) multiplied by SAD (n), the higher the degree of similarity that has been corrected.

When the corrected similarity MSAD (n) of each area is obtained for the histogram shown in FIG.
In area A, the appearance frequency Ha is “2”.
MSAD (A) = SAD (A) × (1− (2/25) × K)
In area B, the appearance frequency Hb is “1”.
MSAD (B) = SAD (B) × (1− (1/25) × K)
In area C, the appearance frequency Hc is “0”.
MSAD (C) = SAD (C) × (1− (0/25) × K)
In area D, the appearance frequency Hd is “12”.
MSAD (D) = SAD (D) × (1− (12/25) × K)
In area E, the appearance frequency He is “8”.
MSAD (E) = SAD (E) × (1− (8/25) × K)
In the area F, the appearance frequency Hf is “0”.
MSAD (F) = SAD (F) × (1− (0/25) × K)
In area G, the appearance frequency Hg is “2”.
MSAD (G) = SAD (G) × (1− (2/25) × K)
In area H, the appearance frequency Hh is “0”.
MSAD (H) = SAD (H) × (1− (0/25) × K)
Area I has an appearance frequency Hi of “0”.
MSAD (I) = SAD (I) × (1− (0/25) × K)

Assuming K = 0.5, the modified similarities are as follows.
MSAD (A) = SAD (A) × 0.96
MSAD (B) = SAD (B) × 0.98
MSAD (C) = SAD (C) × 1
MSAD (D) = SAD (D) × 0.76
MSAD (E) = SAD (E) × 0.84
MSAD (F) = SAD (F) × 1
MSAD (G) = SAD (G) × 0.96
MSAD (H) = SAD (H) × 1
MSAD (I) = SAD (I) × 1.

  In the above example, correction is performed such that the similarity is high even when the appearance frequency is low as in the area A. However, if the appearance frequency is less than the specified value, the correction may not be performed, and the similarity is low. It may be corrected with a value larger than “1”. In addition, the range where the movement is expected to be different from the surrounding movement may be corrected so as to reduce the similarity of the area where the appearance frequency is high.

  In the third embodiment, the candidate extraction unit 7 extracts candidate block pairs, but does not extract evaluation block pairs. Further, as described above, the reliability evaluation unit 8 is not provided, and information indicating the candidate block pair extracted by the candidate extraction unit 7 is supplied to the reference pixel position selection unit 9 as it is, and the reference pixel position selection unit 9 In 9, the pixel at the center position of the candidate block pair extracted by the candidate extracting means 7 is selected as the reference pixel position.

  In the third embodiment, the similarity is corrected in accordance with the appearance frequency of each area detected by the histogram generation means 13, so that the candidate block pair can be selected in consideration of the movement of surrounding pixels. Interpolation can be performed accurately.

Embodiment 4 FIG.
FIG. 15 shows a block diagram of a frame interpolation apparatus according to the fourth embodiment. The illustrated frame interpolating apparatus is generally the same as the frame interpolating apparatus of FIG. 14, but similarly to FIG. 11, the reliability evaluating means 8 is provided, and the output of the histogram generating means 13 is supplied to the reference pixel position selecting means 9. Different in that it is. The reference pixel position selection unit 9 in FIG. 14 operates in the same manner as the reference pixel position selection unit 9 in FIG. Other components denoted by the same reference numerals as those in FIGS. 1, 11, and 14 operate in the same manner as in the first, second, and third embodiments.

  The frame interpolating apparatus shown in FIG. 15 has both the additional features described with respect to the second embodiment and the additional features described with respect to the third embodiment. That is, the similarity correction unit 14 operates in the same manner as described in the third embodiment, and the reference pixel position selection unit 9 is described in the second embodiment based on the histogram generated by the histogram generation unit 13. Works as well.

  The reliability evaluation unit 8 evaluates the reliability in the same manner as in the first and second embodiments by using the output (corrected similarity) of the similarity correction unit 14 instead of the similarity calculated by the similarity correction unit 4. . When calculating the equation (2), the corrected similarity is used as Sc and Se. As Sa, an average value of corrected similarities is used. Based on the evaluation result of the reliability evaluation, the reference pixel position of the interpolation pixel is determined by the reference pixel position selection unit 9 as described in the second embodiment. Pixel values are calculated and an image of the interpolated frame is obtained.

  As described above, in the second and fourth embodiments shown in FIGS. 11 and 15, the reference pixel position is selected by the reference pixel selection unit 9 based on the appearance frequency of each area output from the histogram generation unit 13. Is going.

  In the third and fourth embodiments shown in FIGS. 14 and 15, the similarity of the area representative is corrected by the similarity correction unit 14 based on the appearance frequency of the area output from the histogram generation unit 13. The candidate extraction means 7 extracts candidate block pairs and evaluation block pairs based on the corrected similarity.

  In the second to fourth embodiments, the sampling of the histogram (the count for calculating the appearance frequency) is performed for all the pixels in the detection range Rh around the interpolation pixel, but instead, in the detection range Rh. In this case, sampling may be performed only when the candidate block pair is reliable and interpolated (to be counted for appearance frequency calculation).

  As in the second, third, and fourth embodiments described above, when determining the area of the interpolation block pair, the histogram generation result in the already interpolated range around the pixel position to be interpolated is referred to, so that more accurate There is an effect that an interpolated image is obtained. Further, as in the third and fourth embodiments, there is an effect that an interpolation image with higher reliability can be obtained by correcting the similarity of the representative block pair from the generated histogram.

  The above-described interpolation apparatuses according to the first to fourth embodiments can also be realized by software, that is, by a programmed computer.

  As an application example of the present invention, it can be applied to frame frequency conversion for television and frame frequency conversion for commercial monitors. Even if the frame frequency is not converted, it can be applied to frame conversion in which one redundant frame (a frame having the same content as the immediately preceding frame) is removed and a new frame is generated.

It is a block diagram which shows the interpolation apparatus of Embodiment 1 of this invention. FIG. 3 is a diagram showing a pair of pixel blocks that are sequentially selected for selection of a reference pixel position in the first embodiment. (A)-(d) is a figure which shows the pixel block in the 1st reference frame selected sequentially for selection of the reference pixel position in Embodiment 1. FIG. (A)-(d) is a figure which shows the pixel block in the 2nd reference frame selected sequentially for selection of the reference pixel position in Embodiment 1. FIG. (A)-(d) is a figure which shows the pixel block in the 2nd reference frame selected sequentially for selection of the reference pixel position in Embodiment 1. FIG. It is a figure shown with the matrix which represents roughly the direction and length (distance between center pixel positions) of the straight line which connects the center pixel positions of the pixel block of a plurality of block pairs extracted by block pair extraction means 3. It is a figure which represents roughly the direction and length (distance between center pixel positions) of the straight line which connects the center pixel positions of the pixel block of the block pair corresponding to each element position of the matrix of FIG. It is a figure which shows roughly the direction and magnitude | size of a motion corresponding to the area divided | segmented by the area division means 5. FIG. 6 is a diagram showing pixel blocks in a first reference frame that are sequentially selected for selection of a reference pixel position in the first embodiment. FIG. FIG. 6 is a diagram illustrating pixel blocks in a second reference frame that are sequentially selected for selection of a reference pixel position in the first embodiment. It is a block diagram which shows the interpolation apparatus of Embodiment 2 of this invention. FIG. 10 is a diagram showing a histogram generation range in the second embodiment. It is a figure which shows the example of the histogram production | generation result in each area. It is a block diagram which shows the interpolation apparatus of Embodiment 3 of this invention. It is a block diagram which shows the interpolation apparatus of Embodiment 4 of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Input terminal, 2 Delay means, 3 Block pair extraction means, 4 Similarity calculation means, 5 Area division means, 6 Area representative extraction means, 7 Candidate extraction means, 8 Reliability evaluation means, 9 Reference pixel position selection means, 10 Delay means, 11 symmetrical position interpolation means, 12 output terminals, 13 histogram generation means, 14 similarity correction means.

Claims (21)

A frame interpolation device for generating a pixel value of an interpolation pixel in an interpolation frame between a first reference frame and a second reference frame,
One of the pixel block pair consisting of the pixel block of the first reference frame and the pixel block of the second reference frame that is point-symmetric with respect to the position of the interpolation pixel, and the pixel block pair Block pair extracting means for extracting a plurality of one or both of the pixel block pairs in which the pixel blocks are replaced with pixel blocks shifted by one pixel;
Similarity calculation means for calculating the similarity of two pixel blocks based on the pixel values of the pixels in the two pixel blocks for each of a plurality of pixel block pairs extracted by the block pair extraction means;
The plurality of block pairs extracted by the block pair extraction unit are divided into a plurality of areas such that those in which the directions of straight lines connecting the central pixels of the pixel blocks of each block pair are similar belong to the same area. Area dividing means to divide into
In each area divided by the area dividing means, an area representative extracting means for selecting a pixel block pair having the highest similarity calculated by the similarity calculating means as an area representative block pair;
Among the plurality of area representative block pairs extracted by the area representative extracting unit, a pixel block pair having the largest similarity calculated by the similarity calculating unit or a corrected one is selected as a candidate block pair. In addition, among the plurality of area representative block pairs extracted by the area representative extracting means, the similarity calculated by the similarity calculating means or a correction of the same among the other than the candidate block pairs Candidate block pair extraction means for selecting the pixel block pair having the largest as the evaluation block pair;
A reliability evaluation means for evaluating the reliability of the candidate block pair based on the similarity of the candidate block pair or the corrected version thereof and the similarity of the evaluation block pair or the size of the corrected block pair; and
Reference pixel position selection means for selecting a center position of each pixel block constituting the candidate block pair as a reference pixel position;
Symmetric position interpolation means for generating the interpolation pixel based on the pixel value of the reference pixel position selected by the reference pixel position selection means,
The reference pixel position selection means, when the reliability evaluation means determines that there is reliability, selects the center position of each pixel block constituting the candidate block pair as a reference pixel position, and has no reliability. If it is determined, a frame interpolation device that selects a previously-selected reference pixel position as the reference pixel position of the interpolation pixel.   The apparatus further comprises frame delay means for delaying the selection result of the reference pixel position selection means by one frame, and the reference pixel position selection means uses the output of the frame delay means as the previously selected reference pixel position. The frame interpolation apparatus according to claim 1. The reference pixel position selected by the reference pixel position selection means is set as the central pixel position for each of a plurality of pixels that are positioned around the interpolation pixel and that are to be interpolated before the pixel that is to be interpolated. A histogram generating means for detecting an appearance frequency of an area to which a block pair consisting of pixel blocks belongs,
The reference pixel position selection means determines the center pixel position of the area representative block pair of the area having a high appearance frequency detected by the histogram generation means when the reliability evaluation means evaluates that there is no reliability. The frame interpolation apparatus according to claim 1, wherein the frame interpolation apparatus is used as a selected reference pixel position.   The histogram generation means is subject to counting for obtaining the appearance frequency on the condition that the reliability evaluation means evaluates the reliability and the reference pixel position selection means selects the appearance. The frame interpolation apparatus according to claim 3. According to the appearance frequency of each area detected by the histogram generation means, further comprising a similarity correction means for correcting the similarity of the area representative block pair of each area extracted by the area representative extraction means,
5. The frame interpolation device according to claim 3, wherein the candidate extraction unit selects the candidate block pair and the evaluation block pair based on the similarity corrected by the similarity correction unit. 6. .   6. The frame interpolation apparatus according to claim 5, wherein the similarity correction unit corrects the similarity to a larger value as the appearance frequency is higher.   The reliability evaluation means is the similarity of the area representative block pairs of all the areas extracted by the area representative extraction means or the average value of those corrected, the similarity of the candidate block pairs or the correction thereof, and 2. The frame interpolating apparatus according to claim 1, wherein the reliability is evaluated using the similarity of the evaluation block pair or a similarity obtained by correcting the similarity. The reliability evaluation means includes the following inequality:
{(Se-Sc)} × B + C> (Sa-Se)
(Se represents the similarity of the evaluation block pair or a corrected version thereof,
Sc represents the similarity of the candidate block pair or a corrected version thereof,
Sa represents the similarity of all area representative block pairs or the average of the corrected values,
B represents a coefficient of 1 or more,
C represents a constant of 0 or more)
The frame interpolating apparatus according to claim 7, wherein when the condition is satisfied, it is determined that there is reliability. A frame interpolation device for generating a pixel value of an interpolation pixel in an interpolation frame between a first reference frame and a second reference frame,
One of the pixel block pair consisting of the pixel block of the first reference frame and the pixel block of the second reference frame that is point-symmetric with respect to the position of the interpolation pixel, and the pixel block pair Block pair extracting means for extracting a plurality of one or both of the pixel block pairs in which the pixel blocks are replaced with pixel blocks shifted by one pixel;
Similarity calculation means for calculating the similarity of two pixel blocks based on the pixel values of the pixels in the two pixel blocks for each of a plurality of pixel block pairs extracted by the block pair extraction means;
The plurality of block pairs extracted by the block pair extraction unit are divided into a plurality of areas such that those in which the directions of straight lines connecting the central pixels of the pixel blocks of each block pair are similar belong to the same area. Area dividing means to divide into
In each area divided by the area dividing means, an area representative extracting means for selecting a pixel block pair having the highest similarity calculated by the similarity calculating means as an area representative block pair;
Similarity correction means for correcting the similarity of the area representative block pair selected by the area representative extraction means to generate a correction similarity;
Candidate extraction means for selecting, as a candidate block pair, a pixel block pair having the highest correction similarity among the plurality of area representative block pairs extracted by the area representative extraction means;
Reference pixel position selection means for selecting a center position of each pixel block constituting the candidate block pair as a reference pixel position;
Symmetric position interpolation means for generating the interpolation pixel based on the pixel value of the reference pixel position selected by the reference pixel position selection means;
The reference pixel position selected by the reference pixel position selection means is set as the central pixel position for each of a plurality of pixels that are positioned around the interpolation pixel and that are to be interpolated before the pixel that is to be interpolated. A histogram generating means for detecting the appearance frequency of an area to which a block pair consisting of pixel blocks belongs,
The similarity correction unit corrects the similarity of the area representative block pair of each area extracted by the area representative extraction unit according to the appearance frequency of each area detected by the histogram generation unit. A frame interpolation device. Further provided with a reliability evaluation means,
The candidate extraction unit is configured to evaluate a pixel block pair having the highest correction similarity among the plurality of area representative block pairs extracted by the area representative extraction unit, except for the candidate block pair. Select as a pair,
The reliability evaluation means evaluates the reliability of the candidate block pair based on the correction similarity of the candidate block pair and the magnitude of the correction similarity of the evaluation block pair,
The histogram generation means is subject to counting for obtaining the appearance frequency on the condition that the reliability evaluation means evaluates the reliability and the reference pixel position selection means selects the appearance. The frame interpolation apparatus according to claim 9. The frame interpolation apparatus according to claim 9 or 10, wherein the similarity correction unit corrects the similarity to a larger value as the appearance frequency is higher.   The reliability evaluation unit calculates an average value of correction similarities of area representative block pairs of all areas extracted by the area representative extraction unit, a correction similarity of the candidate block pair, and a correction similarity of the evaluation block pair. The frame interpolation apparatus according to claim 10, wherein reliability is evaluated. The reliability evaluation means includes the following inequality:
{(Se-Sc)} × B + C> (Sa-Se)
(Se represents the corrected similarity of the evaluation block pair,
Sc represents the corrected similarity of the candidate block pair,
Sa represents an average value of the correction similarities of all area representative block pairs,
B represents a coefficient of 1 or more,
C represents a constant of 0 or more)
The frame interpolating apparatus according to claim 12, wherein when it is satisfied, it is determined that there is reliability.   The block pair extraction unit is configured to provide one pixel block for a pixel block pair including a pixel block at a point-symmetrical position with respect to the interpolation pixel and a pixel block pair including a pixel block at the point-symmetrical position. 14. The frame interpolation device according to claim 1, wherein a pixel block pair replaced by a pixel block shifted by one pixel to the left or one pixel to the right is extracted.   The block pair extraction unit is configured to provide one pixel block for a pixel block pair including a pixel block at a point-symmetrical position with respect to the interpolation pixel and a pixel block pair including a pixel block at the point-symmetrical position. 14. The frame interpolation device according to claim 1, wherein a pixel block pair replaced with a pixel block shifted by one pixel or one pixel below is extracted.   The block pair extraction unit is configured to provide one pixel block for a pixel block pair including a pixel block at a point-symmetrical position with respect to the interpolation pixel and a pixel block pair including a pixel block at the point-symmetrical position. 2. A pixel block pair that is replaced with a pixel block that is shifted diagonally right up by one pixel, diagonally down by one pixel right, diagonally up by one pixel left, or diagonally down by one pixel left is extracted. 14. The frame interpolation device according to any one of 1 to 13.   The similarity calculation means obtains the sum of absolute values of differences between pixel values of at least some of the pixels in one pixel block constituting the pixel block pair and pixels at corresponding positions in the other pixel block, 17. The frame interpolating apparatus according to claim 1, wherein the sum is output as representing the degree of similarity.   The similarity calculation means obtains a sum of absolute values of differences between pixel values of all pixels in one pixel block constituting the pixel block pair and pixels at corresponding positions of the other pixel block. The frame interpolation apparatus according to claim 17.   The similarity calculation means calculates a difference between a pixel value of a pixel after performing equal thinning processing on pixels in one pixel block constituting a pixel block pair and a pixel at a corresponding position in the other pixel block. The frame interpolation apparatus according to claim 17, wherein the sum of absolute values of the frame interpolation is obtained. A frame interpolation method for generating a pixel value of an interpolation pixel in an interpolation frame between a first reference frame and a second reference frame,
One of the pixel block pair consisting of the pixel block of the first reference frame and the pixel block of the second reference frame that is point-symmetric with respect to the position of the interpolation pixel, and the pixel block pair A block pair extraction step of extracting a plurality of one or both of the pixel block pairs obtained by replacing the pixel blocks with pixel blocks shifted by one pixel;
A similarity calculation step of calculating the similarity of two pixel blocks based on the pixel values of the pixels in the two pixel blocks for each of a plurality of pixel block pairs extracted by the block pair extraction step;
The plurality of block pairs extracted by the block pair extraction step are divided into a plurality of areas such that those in which the directions of straight lines connecting the central pixels of the pixel blocks of each block pair are similar belong to the same area. Area dividing step to divide into
In each area divided by the area dividing step, an area representative extracting step of selecting a pixel block pair having the highest similarity calculated by the similarity calculating step as an area representative block pair;
Among the plurality of area representative block pairs extracted by the area representative extraction step, a pixel block pair having the largest similarity calculated by the similarity calculation step or a corrected one is selected as a candidate block pair. In addition, among the plurality of area representative block pairs extracted by the area representative extraction step, the similarity calculated by the similarity calculation step or a correction of the similarity among those other than the candidate block pair A candidate block pair extraction step of selecting a pixel block pair having the largest as a block pair for evaluation;
A reliability evaluation step of evaluating the reliability of the candidate block pair based on the similarity of the candidate block pair or a corrected version thereof and the similarity of the evaluation block pair or the size of the corrected block pair; and
A reference pixel position selection step of selecting a center position of each pixel block constituting the candidate block pair as a reference pixel position;
A symmetric position interpolation step for generating the interpolation pixel based on the pixel value of the reference pixel position selected by the reference pixel position selection step, and
In the reference pixel position selection step, when the reliability evaluation step determines that there is reliability, the center position of each pixel block constituting the candidate block pair is selected as a reference pixel position, and there is no reliability. If it is determined, a frame interpolation method, wherein a reference pixel position selected before is selected as the reference pixel position of the interpolation pixel. A frame interpolation method for generating a pixel value of an interpolation pixel in an interpolation frame between a first reference frame and a second reference frame,
One of the pixel block pair consisting of the pixel block of the first reference frame and the pixel block of the second reference frame that is point-symmetric with respect to the position of the interpolation pixel, and the pixel block pair A block pair extraction step of extracting a plurality of one or both of the pixel block pairs obtained by replacing the pixel blocks with pixel blocks shifted by one pixel;
A similarity calculation step of calculating the similarity of two pixel blocks based on the pixel values of the pixels in the two pixel blocks for each of a plurality of pixel block pairs extracted by the block pair extraction step;
The plurality of block pairs extracted by the block pair extraction step are divided into a plurality of areas such that those in which the directions of straight lines connecting the central pixels of the pixel blocks of each block pair are similar belong to the same area. Area dividing step to divide into
In each area divided by the area dividing step, an area representative extracting step of selecting a pixel block pair having the highest similarity calculated by the similarity calculating step as an area representative block pair;
A similarity correction step of generating a corrected similarity by correcting the similarity of the area representative block pair selected in the area representative extraction step;
A candidate extraction step of selecting, as a candidate block pair, a pixel block pair having the highest correction similarity among the plurality of area representative block pairs extracted by the area representative extraction step;
A reference pixel position selection step of selecting a center position of each pixel block constituting the candidate block pair as a reference pixel position;
A symmetric position interpolation step for generating the interpolation pixel based on a pixel value of the reference pixel position selected by the reference pixel position selection step;
The reference pixel position selected in the reference pixel position selection step is set as the central pixel position for each of a plurality of pixels that are positioned around the interpolation pixel and that are to be interpolated before the pixel that is to be interpolated. And a histogram generation step for detecting the appearance frequency of an area to which a block pair consisting of pixel blocks belongs,
In the similarity correction step, the similarity of the area representative block pair of each area extracted in the area representative extraction step is corrected according to the appearance frequency of each area detected in the histogram generation step. Frame interpolation method.

Images