JP5078428B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for performing frame interpolation of a moving image.

  As a technique for improving the flicker and afterimage feeling of a display device, a double speed driving technique for doubling the number of frames per second of a moving image is known. For example, by converting a video of 60 frames per second into a video of 120 frames per second, flicker and afterimage can be greatly improved. The doubling of the number of frames is performed by inserting a new interpolation frame between each frame constituting the original moving image.

  As a method of generating the interpolated frame to be inserted, a frame interpolation technique for generating by a method using a motion vector is known (see, for example, Patent Document 1).

  This interpolation frame generation method will be described with reference to FIG. FIG. 23 is a diagram schematically showing a conventional method for generating an interpolation frame. Here, the current frame is represented as F [0], the frame one frame before (the past frame) is represented as F [−1], and the interpolated frame inserted between these frames is represented as F [−0.5]. .

  In the generation of the interpolation frame, as shown in FIG. 23A, first, the current frame F [0] of the moving image is divided into a plurality of rectangular blocks. Then, block matching is performed between the current frame F [0] and the past frame F [-1], and a motion vector between the current frame F [0] and the past frame F [-1] of each block is obtained. .

Next, as shown in FIG. 23B, a frame F [−0.5] is set as an interpolated frame inserted between the current frame F [0] and the past frame F [−1], and the above calculation is performed. The block of the current frame F [0] is mapped (arranged) on the past frame F [−0.5] according to the motion vector thus determined. In this way, the interpolation frame F [−0.5] inserted between the current frame F [0] and the past frame [−1] is generated.
Japanese Patent Laid-Open No. 03-263890

  However, in the case of the above-described conventional technique, a motion vector cannot be searched, and an area where an image cannot be mapped or an area after moving a moving image portion remains as a blank portion on an interpolation frame. There is a case. In addition, boundary noise due to discontinuity with surrounding blocks may occur near the boundary of the mapped block.

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of generating a high-quality interpolation frame.

To achieve the above SL is provided an image processing apparatus of the present invention is an image processing apparatus that performs frame interpolation of inserting an interpolated frame between a pair of temporally successive frames, one of said pair of frames A motion vector detecting means for dividing the block into a plurality of blocks and detecting a motion vector between the pair of frames for each of the divided blocks; and an area including the block for each block and having a size larger than the size of the block the cut from one of the pair of frames, a region cut out for each of the blocks, and arrangement means for arranging on the interpolation frame in accordance with the detected motion vector of the corresponding block, disposed on the interpolation frame A window function is applied to each of the extracted regions and applied to each of the regions Depending on the value of the window function, and a determining means for determining a pixel value of the overlapping portion between the cut out realm disposed on the interpolated frame, wherein the determining means of the overlapping portion If the maximum value of the window function values for each of the pixels in each region including the target pixel position is greater than or equal to the threshold value, the pixel value of the pixel in the region corresponding to the maximum window function value is the target pixel. If all the window function values for each pixel in each area are smaller than the threshold value, the pixel value of each pixel is weighted using the window function value for each pixel in each area. It calculates an average value, and wherein the pixel values and to Rukoto of the target pixel position weighted average issued the calculated.

To achieve the above SL is provided an image processing method of the present invention is an image processing method for performing a frame interpolation of inserting an interpolated frame between a pair of temporally successive frames, the pair of frame A motion vector detecting step of dividing one into a plurality of blocks and detecting a motion vector between the pair of frames for each of the divided blocks; and a size that includes the block for each block and is larger than the size of the block An area is cut out from one of the pair of frames, and the area cut out for each block is arranged on the interpolation frame according to the detected motion vector of the corresponding block, and the area is arranged on the interpolation frame. A window function is applied to each of the extracted regions and applied to each of the regions Depending on the value of the function, and a determination step of determining a pixel value of the overlapping portion between the cut out realm disposed on the interpolated frame, wherein the determining step, the target pixel of the overlapping portion When the maximum value of the window function values for each of the pixels in each region including the position is greater than or equal to the threshold value, the pixel value of the pixel in the region corresponding to the maximum window function value is When all of the window function values for each of the pixels in each region are smaller than the threshold value, the weighted average value of the pixel values of the respective pixels using the value of the window function for each of the pixels in each region It is calculated and wherein the pixel values and to Rukoto of the target pixel position weighted average issued the calculated.

  According to the present invention, a high-quality interpolation frame can be generated, and a high-quality moving image with less flicker and afterimage can be obtained.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the first embodiment of the present invention.

  As shown in FIG. 1, the image processing apparatus according to the present embodiment includes a frame memory 1, a block matching unit 2, a vector interpolation unit 3, an overlay processing unit 4, a window function table 5, and a normalization unit 6. The block matching unit 2, the vector interpolation unit 3, the superposition processing unit 4, and the normalization unit 6 are configured by a CPU (not shown) executing a program stored in a memory (not shown) such as a ROM. It is what is done. That is, these are configured by software.

  Here, in the following description, the current frame is denoted as F [0], the previous frame is denoted as F [-1], and the interpolated frame inserted between these frames is denoted as F [-0.5]. To do. In addition, a pixel at coordinates (x, y) constituting a frame image is denoted as P (x, y). When specifying a pixel on the frame, it is expressed as F [f]: P (x, y).

  The frame memory 1 is a memory for holding a pair of temporally continuous frames and an interpolation frame inserted between these frames. Specifically, the frame memory 1 holds an image S1 in units of frames constituting a moving image. Here, the frame memory 1 holds an image of the current frame F [0] as a frame constituting a moving image and an image of a past frame F [−1] one frame before that are input from an external device. Has been. In addition, a storage area for the interpolation frame inserted between the current frame F [0] and the past frame F [-1] is secured.

  Next, the block matching unit 2 will be described with reference to FIGS. FIG. 2 is a diagram illustrating an example of a plurality of blocks cut out from the current frame F [0]. FIG. 3 is a diagram illustrating an example of a block cut out from the past frame F [−1]. FIG. 4 is a diagram showing the motion vector S14 detected based on the offset of the partial image (block of 8 × 8 pixels) S3a that most closely matches the block S2.

  In order to obtain a motion vector, the block matching unit 2 divides the image of the current frame F [0] held in the frame memory 1 into a plurality of rectangular blocks S2, and cuts out these blocks S2. The block size of each block S2 is appropriately determined for each apparatus to which the present invention is applied. As the block size increases, the motion vector search accuracy improves. Conversely, as the block size decreases, the processing capability of the apparatus can be reduced, and the apparatus can be configured at low cost.

  In the present embodiment, it is assumed that the block size of the block S2 is 8 × 8 pixels (Bs = 8) as shown in FIG. Here, the block image in the Xth row from the top and the Yth column from the left is denoted as B [X, Y]. For example, the block S2 in the first row from the top is B [1,1], B [2,1], B [3,1],... From the left, and the block S2 in the second row from the top is from the left. B [1,2], B [2,2], B [3,2],.

  Here, when specifying each block on the frame, each block is expressed as F [f]: B [X, Y]. Furthermore, a pixel located at the relative coordinates (i, j) of the block image B [X, Y] is denoted as B [X, Y] (i, j). That is, P (7,9) and B [1,2] (7,1) indicate the same pixel.

  Further, the block matching unit 2 cuts out a block S3 for obtaining a motion vector from the frame F [-1]. The block size of the block S3 is determined by the motion vector search range Sa. The search range Sa is appropriately determined for each system to which the present invention is applied. As the search range Sa becomes wider, a motion vector can be obtained even for a fast-moving moving image. Conversely, as the search range Sa becomes smaller, the processing capability of the device can be reduced, and the device can be configured at low cost.

  In the present embodiment, as shown in FIG. 3, the search range Sa is 12 pixels. That is, 8 + 12 × 2 = 32 obtained by adding the search range Sa × 2 to the block size (Bs) of the block S2 is the size of one side of the block S3, and the block size of the block S3 is 32 × 32 pixels.

  As shown in FIG. 4, the block matching unit 2 searches the block S3 for a partial image (8 × 8 pixel block) S3a that most closely matches the block S2. Then, the offset of the partial image S3a from the position of the block S2 is detected as the motion vector S4.

  Here, for determining whether or not the partial image S3a in the block S3 matches the block S2, the sum of absolute differences of the pixel values d corresponding to the partial image S3a and the block S2 is used. The smaller this value, the higher the degree of matching of images. Specifically, when the block S2 is F [0]: B [X, Y], the block S3 includes F [-1]: P (x-Sa, y-Sa) to F [-1]: This is a block whose diagonal is P (x + Bs + Sa-1, y + Bs + Sa-1). The pixels of the partial image S3a in the block S3 are expressed as F [−1]: B [X, Y] (i + h, j + v) {h = −Sa to + Sa | v = −Sa to + Sa}. .

  Here, the sum of the absolute differences of F [0]: B [x, y] (i, j) and F [-1]: B [X, Y] (i + h, j + v) is D [h, v]. Then, D [h, v] is expressed by the following equation (1).

If the minimum value of D [h, v] is D [H, V],
D [H, V] = min (D [h, v])
H and V satisfying the above are required. Since the vector (H, V) that satisfies this condition is a vector indicating the past frame F [-1] from the current frame F [0],-(H, V) obtained by multiplying this vector by -1 is F [0]: The motion vector S4 in B [x, y].

  Hereinafter, this motion vector S4 is expressed as F [0]: B [x, y]: [S4]. When expressing the elements of the motion vector S4 independently, they are expressed as [S4]: H, [S4]: V.

  Next, the vector interpolation unit 3 will be described with reference to FIG. FIG. 5 is a diagram showing an interpolation motion vector S14 obtained based on the motion vector S4 obtained by the block matching unit 2.

  As shown in FIG. 5, the vector interpolation unit 3 multiplies the motion vector S4 obtained by the block matching unit 2 by -1/2 to indicate an interpolation motion indicating a position on the interpolation frame F [-0.5]. Vector S14 is obtained. The interpolation motion vector S14 is expressed by the following equation (2).

  Next, the superposition processing unit 4 will be described with reference to FIGS. FIG. 6 is a diagram showing the size of an image mapped on the interpolation frame F [−0.5] by the overlay processing unit 4. FIG. 7 is a diagram showing a configuration of fields that store the addition / maximum value mode, the influence degree, and the pixel value, which are associated with each interpolation pixel. FIG. 8 is a diagram showing the distribution of the window function ω. FIG. 9 is a diagram showing characteristics representing the relationship between the motion vector of the window function ω and the influence level. FIG. 10 is a diagram schematically showing the relationship between the interpolation motion vector S14, the mapped image, and the influence S16.

  The superposition processing unit 4 performs superposition processing for mapping each block of the current frame F [0] onto the interpolation frame F [-0.5]. In this superposition processing, as shown in FIG. 6, first, an area (shaded portion) that includes the block S2 (dotted line portion) corresponding to the start point of the interpolated motion vector S14 and is larger than the block size of the block S2 is included. Cut out from the current frame F [0]. Then, the image of the clipped area is mapped (arranged) to the solid line area on the interpolation frame F [−0.5] according to the interpolation motion vector S14.

  Here, in order to prevent a gap portion from being generated after mapping, it is desirable that the area (shaded portion) cut out from the frame F [0], that is, the size Is of the image to be mapped is larger than the block size of the block S3. In this embodiment, Is = 48.

Next, the pixel S22 in the segmented area of the current frame F [0] is sequentially read from the frame memory 1. The readout range of the pixel S22 is as follows.
{Q = x + (Bs-Is) / 2 to x + (Bs + Is) / 2-1, r = y + (Bs-Is) / 2 to y + (Bs + Is) / 2}
The read pixel S22 is expressed as [S22] = F [0]: P (q, r).

Next, in order to perform the overlapping process with the pixel S22 in the region cut out from the current frame F [0], the interpolation pixel S15 from the interpolation frame F [−0.5] in the frame memory 1 is performed according to the interpolation motion vector S14. Is read out. The pixel S15 to be read is
[S15] = F [−0.5]: P (q + [V11]: H, r + [V11]: V)
It becomes.

Here, as shown in FIG. 7, each field of the [addition / maximum value] field, the [degree of influence] field, and the [pixel value] field corresponds to the interpolation pixel S15 of the interpolation frame F [−0.5]. It is attached. Here, the [sum / Maximum Value field, flag interpolation pixel mode indicating whether the maximum value mode or an addition mode is stored.

  In the present embodiment, as will be described later, the window function ω is applied to each of the clipped regions arranged on the interpolation frame F [−0.5]. Then, either the addition mode or the maximum value mode is selected according to the value of the window function ω applied to each of the regions and the threshold value ψ, and the interpolation frame F [−0.5 is selected according to the selected interpolation pixel mode. ] The pixel values of the overlapping portions between the regions arranged above are calculated and adaptively determined.

  Specifically, the maximum value of the values of the window function ω for each pixel in each clipped region at the overlapping point of interest (attention pixel position) is greater than or equal to the threshold ψ (greater than or equal to the threshold). In this case, the maximum mode is selected, and the pixel value of the pixel corresponding to the maximum window function ω is set as the pixel value of the target point. On the other hand, when all the values of the window function ω for each of the pixels in each clipped region at the overlapping point of interest (the pixel position of interest) are smaller than the threshold ψ (when they are within a predetermined range), The addition mode is selected. In this addition mode, the weighted average value of the pixel values of the respective pixels is obtained by using the value of the window function ω for each of the pixels in each of the cut out regions at the overlapping point of interest (attention pixel position). Calculated. Then, the calculated weighted average value is set as the pixel value of the attention point.

  When the flag of the [addition / maximum value] field indicates the addition mode, the sum of influence degrees of all the pixels superimposed on the pixel position of the interpolation pixel is stored in the [influence degree] field. The [pixel value] field stores the sum of (pixel value × influence degree) of all the pixels superimposed on the pixel position of the interpolation pixel.

  On the other hand, when the flag in the [addition / maximum value] field indicates the maximum value mode, the [influence degree] field has the maximum influence degree among all the pixels superimposed on the pixel position of the interpolation pixel. The influence degree of the pixel is stored. In the [Pixel value] field, the pixel value of the pixel having the maximum degree of influence among all the pixels superimposed on the pixel position of the interpolation pixel is stored.

  As will be described later, when the creation of an interpolation frame is started, the values of these fields are set such that the influence degree = 0 and the pixel value = 0 when the image of the interpolation frame stored in the frame memory is cleared. Cleared.

  In the following description, the [addition / maximum value] field of the interpolation pixel S15 is represented as [S15]: M, the [influence] field is represented as [S15]: E, and the [pixel value] field is represented as [S15]: P. Shall.

  The superposition processing unit 4 reads from the window function table 5 the influence S16 associated with the pixel at the pixel position (attention point) where the superposition processing is performed. The influence degree S16 is a value of the window function ω stored in the window function table 5. The window function ω is a continuous function having a distribution as shown in FIGS. 8 and 9, for example. Further, as shown in FIG. 8, the window function ω has a sufficiently large influence in a range covering the block corresponding to the start point of the motion vector, and the influence degree becomes 0 as the distance from the block corresponding to the start point of the motion vector increases. Is a function having a characteristic approaching. The domain of the window function ω is equal to the size Is of the image to be mapped (the cropped region) and is ω (x) [x = −Is / 2 to Is / 2].

  For example, as shown in FIG. 10, the value of the window function ω is applied to the pixel at the point of interest of the image mapped on the interpolation frame F [−0.5] corresponding to the reference pixel of the current frame F [1]. Then, the pixel value of the attention point is determined according to the value of the window function ω, that is, the influence S16.

  In the addition mode, the overlap resolution process is switched according to whether or not the value of the influence S16 of the corresponding interpolation pixel S15 is equal to or greater than the threshold ψ. Here, the threshold ψ is a value used for switching the subsequent overlap resolution processing. When the threshold value ψ is increased, the area where the weighted average of pixels is increased, and a smooth image can be obtained. When the threshold value ψ is reduced, the area where the weighted average of pixels is reduced and a sharp image is obtained. Normally, the threshold ψ is set to a value at which the block size of the block for calculating the motion vector S4 and the switching point of the overlap resolution process are substantially the same.

  When the influence degree S16 is not equal to or greater than the threshold ψ, the value of the read influence degree S16 is added to the value (influence degree) of the [influence degree] field of the interpolation pixel S15. Further, a value obtained by multiplying the pixel value of the pixel S22 and the influence level S16 is added to the value (pixel value × the influence level) of the [pixel value] field of the interpolation pixel S15. That is, the calculation according to the following equation (3) is performed, and the weighted average value of the pixels is calculated.

The interpolation pixel S15 in which the value of the corresponding field is rewritten in this way is stored in a register (not shown) as the interpolation pixel S17. Then, the interpolation pixel S17 held in this register is overwritten at the corresponding position of the interpolation frame F [−0.5] in the frame memory 1. That is,
F [−0.5]: P (i, j) = [S17]
The operation is performed.

  On the other hand, when the influence S16 is equal to or greater than the threshold ψ, the flag in the [addition / maximum value] field of the interpolation pixel S15 is set as the flag in the [maximum value] mode. Then, the value of the [influence] field of the interpolation pixel S <b> 15 is rewritten to the value of the influence S <b> 16 read from the window function table 5. Further, the value of the [pixel value] field of the interpolation pixel S15 is rewritten to the pixel value of the reference image I2. That is, the calculation according to the following equation (4) is performed.

In this case, similarly, the interpolated pixel S15 in which the value of the corresponding field is rewritten is stored in the register (not shown) as the interpolated pixel S17, and then the interpolated frame F [−0.5] in the frame memory 1 is stored. The corresponding position above is overwritten. That is,
F [−0.5]: P (i, j) = [S17]
The operation is performed.

  In the maximum value mode, the value of the influence S16 read from the window function table 5 is compared with the value of the [influence] field of the interpolation pixel S15. Here, if the value of the influence degree S16 is equal to or greater than the value of the [influence degree] field of the interpolation pixel S15, the value of the [influence degree] field of the interpolation pixel S15 is the influence degree S16 read from the window function table 5. Is rewritten to the value of. Further, the value of the [pixel value] field of the interpolation pixel S15 is rewritten to the pixel value of the pixel S22 of the block S2. That is, the calculation according to the above equation (4) is performed. Similarly, the interpolated pixel S15 in which the value of the corresponding field is rewritten is stored in the register (not shown) as the interpolated pixel S17, and then on the interpolated frame F [−0.5] of the frame memory 1. The corresponding position is overwritten.

  On the other hand, if the value of the influence S16 is not equal to or greater than the value of the [influence] field of the interpolation pixel S15, the interpolation pixel S15 is stored in the register (not shown) as the interpolation pixel S17, and then the frame memory 1 The corresponding position on the interpolation frame F [−0.5] is overwritten.

  Thus, the processing for one block of each block S2 of the current frame F [0] is completed. When the processing is completed for all the blocks S2 of the current frame F [0], the normalization output process by the normalization unit 6 is performed. On the other hand, when the processing for all the blocks S2 of the current frame F [0] is not completed, the next block S2 is cut out from the current frame F [0] by the block matching unit 2 again.

  Next, the normalization unit 6 will be described.

  The normalizing unit 6 sequentially reads out the interpolation pixel S <b> 17 constituting the interpolation frame F [−0.5] from the frame memory 1. The read range is {x = 1 to X, y = 1 to Y}.

  The normalizing unit 6 refers to the flag of the [addition / maximum value] field of the read interpolation pixel S17, and when the flag indicates the addition mode, the normalization unit 6 sets the value of the [pixel value] field of the interpolation pixel S17 to [influence]. Divide by field value to normalize pixel values. By this normalization, the value of the [pixel value] field is rewritten to the normalized pixel value. Then, the interpolation pixel S17 in which the value of the [pixel value] field is rewritten to the normalized pixel value is output as the normalized pixel S19. That is, the operation represented by the following equation (5) is performed.

  On the other hand, when the flag in the [addition / maximum value] field of the read interpolation pixel S17 indicates the maximum value mode instead of the addition mode, the read interpolation pixel S17 is subjected to the above normalization. Without being output as a normalized pixel S19.

  The normalizing unit 6 repeats the above processing until all of the interpolation pixels constituting the image of the interpolation frame F [−0.5] are output.

  Next, interpolation frame generation processing of the image processing apparatus will be described with reference to FIGS. FIG. 11 is a flowchart showing the procedure of motion vector detection processing by the block matching unit 2 and interpolation vector output processing by the vector interpolation unit 3. FIG. 12 is a flowchart showing the procedure of overlay processing by the overlay processor 4. FIG. 13 is a flowchart showing the procedure of normalization processing by the normalization unit 6.

  In this image processing apparatus, as shown in FIG. 11, first, an interpolated frame F [−0.5] inserted between the current frame F [0] and the past frame F [−1] in the frame memory 1 is displayed. The value of the image storage area is cleared (step S101). Then, the image of the current frame F [0] is input from the external device and held in the frame memory 1 (step S102). At this time, the frame memory 1 holds an image of the past frame F [−1] one frame before the current frame F [0].

  Next, motion vector detection processing by the block matching unit 2 is started. In this motion vector detection process, first, the current frame F [0] held in the frame memory 1 is divided into a plurality of rectangular blocks S2, and these blocks S2 are cut out (step S103). Subsequently, the block S3 is cut out from the past frame F [-1] (step S104). Then, block matching for detecting the motion vector S4 is performed (step S105). In this block matching, a partial image (8 × 8 pixel block image) S3a that most closely matches the block S2 is searched from the block S3. Then, an offset (direction and amount of movement) of the searched partial image S3a from the position of the block image S2 is detected as a motion vector S4.

  Next, the vector interpolation unit 3 multiplies the motion vector S4 detected by the block matching unit 2 by -1/2 to obtain an interpolation motion vector S14 indicating the interpolation frame F [-0.5] (step S106). .

  In this way, when the interpolated motion vector S14 for one block S2 of the current frame F [0] is obtained, the overlay processing by the overlay processing unit 4 is started.

  In the superimposition process, as shown in FIG. 12, first, as an image to be mapped on the interpolation frame F [−0.5], an area cut out from the current frame F [0] (including the block S2) is included. Area S) pixels S22 are sequentially read (step S107). Subsequently, in order to perform an overlapping process with the pixel S22 read from the current frame F [0], the interpolation frame S [15] from the interpolation frame F [−0.5] in the frame memory 1 is performed according to the interpolation motion vector S14. Is read (step S108). Then, the degree of influence S16 (the value of the window function ω stored in the window function table 5) corresponding to the pixel position on which the overlay process is performed is read from the window function table 5 (step S109).

  Next, it is determined whether or not the flag in the [addition / maximum value] field of the interpolation pixel S15 indicates the addition mode (step S110). Here, when the flag indicates the addition mode, it is determined whether or not the read influence S16 is equal to or greater than a threshold value ψ (step S111). When the influence S16 is not equal to or greater than the threshold ψ, the read influence S16 is added to the value (influence) in the [influence] field of the interpolation pixel S15 (step S114). Further, a value obtained by multiplying the pixel value of the pixel S22 and the influence level S16 is added to the value (pixel value × the influence level) of the [pixel value] field of the interpolation pixel S15. That is, the calculation according to the above equation (3) is performed. In this way, the interpolation pixel S15 whose field value has been rewritten is temporarily stored in the register (not shown) as the interpolation pixel S17 (step S116).

  In contrast, if it is determined in step S111 that the influence S16 is equal to or greater than the threshold ψ, a flag indicating the maximum value mode is set for the [addition / maximum value] field of the interpolation pixel S15 (step S111). S112). That is, switching from the addition mode to the maximum value mode is performed. Then, the value of the [influence] field of the interpolation pixel S15 is rewritten to the value of the influence S16 read from the window function table 5 (step S115). Further, the value of the [pixel value] field of the interpolation pixel S15 is rewritten to the pixel value of the corresponding pixel S22. That is, the calculation according to the above equation (4) is performed. In this way, the interpolation pixel S15 whose field value has been rewritten is temporarily stored in the register (not shown) as the interpolation pixel S17 (step S116).

When the flag in the [addition / maximum value] field of the interpolation pixel S15 does not indicate the addition mode in step S110, the mode is the maximum value mode. In this case, it is determined whether or not the value of the influence S16 read from the window function table 5 is equal to or greater than the value of the [influence] field of the interpolation pixel S15 (step S113). Here, if the value of the influence S16 is equal to or greater than the value of the [influence] field of the interpolation pixel S15, the value of the [influence] field of the interpolation pixel S15 is read from the window function table 5. The value is rewritten to the value of S16 (step S115). Further, the value of the [pixel value] field of the interpolation pixel S15 is rewritten to the pixel value of the corresponding pixel S22. That is, the calculation according to the above equation (4) is performed. In this way, the interpolation pixel S15 whose field value has been rewritten is temporarily stored in the register (not shown) as the interpolation pixel S17 (step S116).

  On the other hand, when it is determined in step S113 that the value of the influence S16 is not greater than or equal to the value of the [influence] field of the interpolation pixel I15, the interpolation pixel S15 is stored in a register (not shown) as the interpolation pixel S17. It is temporarily stored (step S116).

  After the interpolation pixel S15 is temporarily stored in the register as the interpolation pixel S17, it is determined whether or not there is a next pixel to be processed in the cut out region (step S117). Here, when there is a next pixel to be processed in the cut out region, the next pixel is overlapped (steps S107 to S116).

  If it is determined in step S117 that there is no next pixel to be processed in the clipped area, the interpolation pixel S17 held in the register is the interpolation frame F [−0.5] in the frame memory 1. The corresponding position above is overwritten (step S118).

  Next, it is determined whether or not there is a next block S2 to be processed in the current frame F [0] (step S119). If there is a next block image S2 to be processed in the current frame F [0], block S3 block matching of the past frame F [-1] corresponding to the next block S2 is performed (steps S103 to S103). Step S118).

  On the other hand, when it is determined in step S119 that there is no next block S2 to be processed in the current frame F [0], normalization processing by the normalization unit 6 is started.

  In the normalization process of the normalization unit 6, as shown in FIG. 13, each interpolation pixel S17 constituting the image of the interpolation frame F [−0.5] is sequentially read from the frame memory 1 (step S120). Then, it is determined whether or not the flag in the [addition / maximum value] field of the read interpolation pixel S17 indicates the addition mode (step S121). Here, when the flag of the [addition / maximum value] field of the read interpolation pixel S17 indicates the addition mode, the value of the [pixel value] field of the interpolation pixel S17 is divided by the value of the [influence] field to obtain a pixel value. Is normalized (step S122). By this normalization, the value of the [pixel value] field of the interpolation pixel S17 is rewritten to the normalized pixel value. Then, the interpolated pixel S17 whose pixel value is rewritten with the normalized pixel value is output as the normalized pixel S19 (step S123).

  On the other hand, when the flag of the [addition / maximum value] field of the read interpolation pixel S17 does not indicate the addition mode, that is, indicates the maximum value mode, the read interpolation pixel S17 is subjected to the above normalization. Instead, it is output as a normalized pixel S19 (step S123).

  Next, it is determined whether or not output of all the interpolation pixels constituting the image of the interpolation frame F [−0.5] has been completed. Here, if the output of all the interpolation pixels constituting the image of the interpolation frame F [−0.5] has not been completed, the normalization unit 6 repeats the processing from step S120. On the other hand, when all the output of the interpolation pixels constituting the image of the interpolation frame F [−0.5] is finished, the normalization unit 6 finishes this process.

  In this way, an image of the interpolated frame [−0.5] inserted between the current frame F [0] and the past frame F [−1] is generated.

  The result of the interpolation frame generation process will be described with reference to FIG. FIG. 14 is a diagram schematically showing an example of the result of the interpolation frame generation processing by the image processing apparatus of FIG.

  By the interpolation frame generation processing, for example, a processing result as shown in FIG. 14 is obtained. Here, in FIG. 14, the positions indicated by the interpolation motion vector S14 are a, b, c, and d. In addition, each of the regions 22a to 22d represented by the dotted frame around the positions a, b, c, and d is a definition region of the window functions ωa to ωd applied to the respective regions (cutout regions). And Further, the influence degree of the window functions ωa to ωd applied to the respective regions 21a to 21d represented by the solid line frames centering on the respective positions a, b, c, and d is larger than the threshold ψ. Suppose that it is an area.

  The sizes of the areas 21a to 21d represented by the solid line frames are equal to the block size Bs of the block S2 in the block matching, and these areas 21a to 21d correspond to areas where images are mapped by a conventional method. The regions outside the respective regions 21a to 21d represented by the solid line frames correspond to regions that are blank without being mapped in the conventional method.

  In the present embodiment, since the value of the window function ω is larger than the threshold value ψ in the region close to the position indicated by the interpolation motion vector S14, the interpolation pixel mode becomes the maximum value mode, and the block S2 corresponding to the closest motion vector S4. Are output without being mixed. As a result, an image near the position indicated by the interpolation motion vector S14 is output clearly.

  In a region far from the position indicated by the interpolation motion vector S14, the value of the window function ω is smaller than the threshold value ψ, so that the interpolation pixel mode is the addition mode, and the pixel values of the block S2 corresponding to all the surrounding motion vectors S4 are Output as weighted average. That is, an image larger than the block S2 corresponding to the motion vector S4 protrudes into an area where the motion vector S4 does not exist and is mapped. Thereby, a blank area does not occur in the interpolation frame. In addition, partial images corresponding to different motion vectors S4 in the region are smoothly connected by the effect of the weighted average, and the generation of boundary noise is suppressed.

  As described above, according to this embodiment, it is possible to generate a high-quality interpolated frame that has no blank portion and suppresses the generation of boundary noise around the moving object, and has high image quality with less flicker and afterimage. Can be obtained.

  Although this embodiment has a high frequency of memory access, it can perform sequential processing, and is therefore configured for software processing. This makes it possible to configure the device without incurring high costs.

  In the present embodiment, the case of performing the double conversion from 60 frames per second to 120 frames per second has been described. However, in addition to the double conversion, frame conversion such as triple conversion and quadruple conversion can be performed. It can be realized by the same procedure. Also, non-integer multiple conversion, such as conversion from 60 frames per second to 90 frames per second, can be realized by the same procedure.

  In the present embodiment, the block matching unit 2, the vector interpolation unit 3, the superposition processing unit 4, and the normalization unit 6 are configured by software, but these can also be configured by hardware. .

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIGS. FIG. 15 is a block diagram showing a configuration of an image processing apparatus according to the second embodiment of the present invention. FIG. 16 is a flowchart showing a procedure of motion vector detection processing by the image processing apparatus of FIG. FIG. 17 is a flowchart showing the procedure of vector extraction processing by the image processing apparatus of FIG. FIG. 18 is a flowchart showing the procedure of the overlap resolution process by the image processing apparatus of FIG. FIG. 19 is a diagram schematically showing a method of selecting candidate vectors by the motion vector selection unit 13 of FIG.

  The image processing apparatus according to the present embodiment is configured by hardware in order to reduce the frequency of memory access and perform parallel processing. Thereby, a high-performance image processing apparatus can be configured.

  As shown in FIG. 15, the image processing apparatus includes a frame memory 1, a motion vector detection unit 12, a motion vector selection unit 13, a distance calculation unit 14, and an influence calculation unit 15. The image processing apparatus also includes a candidate vector holding unit 16, an influence degree determination unit 17, an influence degree normalization unit 18, a weighted average processing unit 19, and a selection unit 20.

  The motion vector detection unit 12 divides the current frame F [0] held in the frame memory 1 into a plurality of blocks S2, and cuts out each block S2. Then, block matching with the block S3 extracted from the past frame F [-1] is performed for each cut out block S2, and between the current frame F [0] and the past frame [-1] for each block S2. Motion vector S4 is detected. That is, the motion vector detection unit 12 detects motion vectors S4 corresponding to the number of divided blocks S2 (for one screen).

  The motion vector selection unit 13 has a memory (not shown), and holds all the motion vectors S4 (motion vector S4 for one screen) detected by the motion vector detection unit 12 in the memory. The motion vector selection unit 13 sequentially selects, from the memory as a candidate vector S5, a motion vector S4 that may affect the pixel of interest from among the motion vectors S4 for one screen held in the memory. And output.

  Selection of this candidate vector S5 will be described. As illustrated in FIG. 19, when the block a is an original block that performs block matching at the α point, the motion vector search range for the block a is a search range b indicated by a one-dot chain line. As a result, the maximum range of detected motion vectors is the range c. Since this motion vector is a vector from the past frame F [−1] to the current frame F [0], when this is mapped to the interpolation frame F [−0.5], the maximum range that the motion vector can take is maximum. The range d is half of the range c. Here, when the motion vector becomes the maximum and points to the β point, the range in which the window function ω centering on the β point can affect is the range e. Similarly, when the motion vector points to the β ′ point, the range that can be influenced by the window function ω is the range e ′.

  That is, the range on the interpolation frame F [−0.5] that may be affected by the motion vector at the point α on the current frame F [0] is the range f. The range f is expressed as [search range] / 2 + [window function definition range].

  Here, focusing on the γ point on the interpolation frame F [−0.5], the motion vector on the current frame F [0] that may affect the γ point is the motion vector existing in the block in the range g. It can be seen that it is.

Therefore, the motion vector S4 existing in [search range] / 2 + [window function definition range] with respect to the target point (target pixel position) is sequentially selected as the candidate vector S5. The search range is
{i = x-Sa / 4-Is / 2 to x + Sa / 4 + Is / 2, j = y-Sa / 4-Is / 2 to y + Sa / 4 + Is / 2}
It becomes. Here, Sa is the motion vector search range, and Is is the size of the image to be mapped, that is, the region cut out from the same current frame F [1] as in the first embodiment.

  The distance calculation unit 14 multiplies the absolute value of the candidate vector S5 output from the motion vector selection unit 13 by -1/2, and points of interest based on the coordinates indicated by the vector on the interpolation frame F [-0.5]. The distance S6 is calculated. Specifically, the following equation (6) is used to calculate the distance S6 to the attention point.

  The influence degree calculation unit 15 holds a window function table (not shown), and calculates an influence degree S7 corresponding to the distance S6 to the point of interest calculated by the distance calculation unit 14 with reference to the window function table. To do.

  The candidate vector holding unit 16 holds the candidate vector S5 output from the motion vector selection unit 13 and the influence S7 calculated by the influence calculation unit 15 in association with each other.

  The influence degree determination unit 17 extracts the candidate vector S5 having the maximum influence degree from all the candidate vectors S5 held in the candidate vector holding part 16 as the maximum motion vector S8. Then, the degree of influence determination unit 17 determines whether or not the degree of influence of the extracted maximum motion vector S8 is greater than or equal to the threshold ψ, outputs this determination result as the degree of influence determination result S9, and extracts the extracted maximum motion The vector S8 is output.

  The influence normalization unit 18 reads out a vector group including all candidate vectors S5 held in the candidate vector holding unit 16 as a candidate vector group S20. Then, the influence degree normalization unit 18 normalizes the influence degree of each candidate vector S5 of the candidate vector group S20, and generates a normalized candidate vector group S11 including all candidate vectors S5 whose influence degree is normalized.

  Here, the total number of candidate vectors included in the candidate vector group S20 is N, the nth candidate vector of the candidate vector group S20 is [S20 (n)], and the degree of influence is [S20 (n)]: E. . If the nth vector of the normalization candidate vector group S11 is [S11 (n)] and the normalization influence is [S11 (n)]: E ′, the normalization influence is [S11 (n)]. ]: E ′ is represented by the following equation (7).

  The weighted average processing unit 19 reads the reference pixel group S21 indicated by the motion vector of the normalization candidate vector group S11 output from the influence degree normalization unit 18 from the frame memory 1. Then, the weighted average processing unit 19 performs weighted average processing of the reference pixel group S21 based on the influence degree of the normalization candidate vector group S11, and uses the weighted average value obtained by the weighted average processing as a pixel value. The pixel S12 is output. Specifically, the weighted average process is performed according to the following equation (8).

  The selection unit 20 selects either the pixel value of the maximum influence pixel S10 or the pixel value of the weighted average pixel S12 based on the influence degree determination result S9 output from the influence degree determination unit 17, and the selected pixel value Are output as the output pixel S13. Specifically, when the influence degree determination result S9 indicates that the influence degree of the maximum motion vector S8 is greater than or equal to the threshold ψ, the maximum influence pixel S10 indicated by the maximum motion vector S8 is the current frame F of the frame memory 1. Extracted from [0], the pixel value of the maximum affected pixel S10 is selected as the pixel value of the output pixel S13. In contrast, when the influence degree determination result S9 indicates that the influence degree of the maximum motion vector S8 is smaller than the threshold ψ, the pixel value of the weighted average pixel S12 output from the weighted average processing unit 19 is the pixel of the output pixel S13. A value is selected.

  Next, interpolation frame generation processing of the image processing apparatus will be described with reference to FIGS.

  In this image processing apparatus, as shown in FIG. 16, first, motion vector detection processing by the motion vector detection unit 12 is started. In the motion vector detection process, the image S1 of the current frame F [0] is input from the external device to the frame memory 1 as a frame constituting the moving image (step S201). At this time, the frame memory 1 holds an image of the past frame F [−1]. In addition, the value of the image storage area of the interpolation frame F [−0.5] in the frame memory 1 is cleared.

  Next, the current frame F [0] held in the frame memory 1 is divided into a plurality of rectangular blocks S2, and one block S2 is cut out from these blocks S2 (step S202). Subsequently, a block S3 corresponding to the extracted one block S2 is cut out from the past frame F [-1] (step S203). Then, block matching between the extracted block S2 and block S3 is performed, and a motion vector S4 is detected (step S204). This block matching is the same as that in the first embodiment. The detected motion vector S4 is output to the motion vector selection unit 13 and held in the memory of the motion vector selection unit 13 (step S205).

  Next, the motion vector detection unit 12 determines whether or not the detection of the motion vector S4 for one screen has been completed (step S206). If the detection of the motion vector S4 for one screen has not been completed, the process returns to step S202, and the next motion vector S4 is detected.

  On the other hand, when the detection of the motion vector S4 for one screen is completed, the candidate vector extraction process is started. At this time, the memory of the motion vector selection unit 13 holds motion vectors S4 corresponding to the number of divided blocks S2 (one screen). The candidate vector extraction process is performed for each pixel constituting the image of the interpolation frame F [−0.5]. The processing range is {x = 1 to X, y = 1 to Y}.

  In the candidate vector extraction process, as shown in FIG. 17, first, the motion vector selection unit 13 sequentially selects candidate vectors S5 from the motion vectors S4 for one screen held in the memory. (Step S207). Then, the distance calculation unit 14 calculates the distance S6 between the position indicated by the vector obtained by multiplying the selected candidate vector S5 by -1/2 and the point of interest using the equation (6) (step S6). S208).

  Next, the influence degree calculation unit 19 calculates an influence degree S7 corresponding to the distance S6 with reference to the window function table 5 (step S209). The calculated influence S7 is stored in the candidate vector storage unit 16 in association with the candidate vector S5 output from the motion vector selection unit 13 (step S210).

  Next, it is determined whether there is a candidate vector S5 to be selected next (step S211). If there is a candidate vector S5 to be selected next, the process returns to step S207 again, and the motion vector selection unit 13 selects the next candidate vector S5. On the other hand, if there is no candidate vector S5 to be selected next, the overlap resolution process is started.

  In the overlap resolution process, as shown in FIG. 18, the influence vector determination unit 17 determines that the candidate vector S5 having the maximum influence level is the maximum motion vector from among the candidate vectors S5 held in the candidate vector holding unit 16. Extracted as S8 (step S212). Then, the influence degree determination unit 17 determines whether or not the influence degree of the extracted maximum motion vector S8 is greater than or equal to the threshold value ψ (step S213). This determination result is output to the influence degree normalization unit 18 and the selection unit 20 as the influence degree determination result S9.

  Here, when the influence degree determination result S9 indicates that the influence degree of the maximum motion vector S8 is smaller than the threshold ψ, the influence degree normalization unit 18 normalizes the influence degree of the candidate vector S5 (step S215). . By this normalization, a normalization candidate vector group S11 is generated and output to the weighted average processing unit 19.

  Next, the weighted average processing unit 19 performs weighted average processing of the reference pixel group S21 based on the degree of influence of the normalization candidate vector group S11 output from the influence degree normalizing unit 18 (step S216). The reference pixel group S21 is indicated by the motion vector of the normalization candidate vector group S11, and the weighted average pixel S12 obtained by the weighted average process of the reference pixel group S21 is output to the selection unit 10. Then, the selection unit 10 selects and outputs the weighted average pixel S12 output from the weighted average processing unit 19 as the output pixel S13 (step S217).

  When the influence degree determination result S9 indicates that the influence degree of the maximum motion vector S8 is equal to or greater than the threshold ψ, the selection unit 20 causes the maximum influence pixel S10 indicated by the maximum motion vector S8 to be the current frame F of the frame memory 1. Extracted from [0]. The maximum influence pixel S10 is selected and output as the output pixel S13 (step S217). In this case, the influence normalization unit 18 does not operate.

  Next, it is determined whether or not the output of all the pixels on the interpolation frame F [−0.5] has been completed (step S218). If the output of all the pixels on the interpolation frame F [−0.5] has not been completed, the process returns to step S217, and the vector extraction process for the next pixel on the interpolation frame F [−0.5] is performed. Is started. On the other hand, when the output of all the pixels on the interpolation frame F [−0.5] is finished, this processing is finished.

  As described above, according to the present embodiment, the same effect as in the first embodiment can be obtained.

(Third embodiment)
Next, a third embodiment of the present invention will be described with reference to FIGS. FIG. 20 is a block diagram showing a configuration of an image processing apparatus according to the third embodiment of the present invention. FIG. 21 is a flowchart showing a procedure of interpolation frame generation processing by the image processing apparatus of FIG. FIG. 22 is a diagram schematically showing the relationship between the mixture ratio S31 calculated by the mixture ratio calculator 21 of FIG. 20 and the degree of influence of the maximum motion vector S8.

  As in the second embodiment, the present embodiment is configured by hardware in order to reduce the frequency of memory access and to perform parallel processing. However, the present embodiment is configured such that, in the overlap resolution processing, the processing is continuously performed without switching the processing using the threshold ψ as in the second embodiment. This is different from the second embodiment.

  As shown in FIG. 20, the image processing apparatus according to the present embodiment includes a frame memory 1, a motion vector detection unit 12, a motion vector selection unit 13, a distance calculation unit 14, and an influence calculation unit 15. The image processing apparatus also includes a candidate vector holding unit 16, a mixture ratio calculation unit 21, an influence normalization unit 18, a weighted average processing unit 19, and a mixing unit 22. Here, the same functional blocks as those of the second embodiment are denoted by the same reference numerals, and description thereof is omitted. And a different part is demonstrated.

  The mixture ratio calculation unit 21 extracts the candidate vector S5 having the maximum influence from all the candidate vectors S5 held in the candidate vector holding unit 16 as the maximum motion vector S8. Then, the mixture ratio calculation unit 21 calculates the mixture ratio S31 of the pixel value of the weighted average pixel S12 with respect to the pixel value of the maximum influence pixel S10 based on the influence degree of the extracted maximum motion vector S8 and the threshold value ψ.

  As shown in FIG. 22, the mixing ratio S31 is obtained when the influence of the maximum motion vector S8 is within a buffer section (predetermined range) having a width of ± buffer value ξ centered on the threshold ψ, and the maximum motion vector S8. It is calculated separately when the degree of influence is outside the buffer interval.

  Specifically, when the influence of the maximum motion vector S8 is within the buffer section, that is, between the lower limit value (ψ−ξ) and the upper limit value (ψ + ξ) of the buffer section, the mixing ratio S31 is: It is calculated so as to gradually increase in proportion to the influence degree of the maximum motion vector S8. Here, when the influence degree of the maximum motion vector S8 is equal to the lower limit value (ψ−ξ) of the buffer section, the mixture ratio S31 is 0%, and the influence degree of the maximum motion vector S8 is the upper limit value (ψ + ξ) of the buffer section. ) Is 100%.

  On the other hand, when the influence degree of the maximum motion vector S8 is outside the buffer section, the influence of the maximum motion vector S8 is when the influence degree of the maximum motion vector S8 is smaller than the lower limit value (ψ−ξ) of the buffer section. The degree may be larger than the upper limit (ψ + ξ) of the buffer section. When the influence degree of the maximum motion vector S8 is smaller than the lower limit value (ψ−ξ) of the buffer section, the mixture ratio S31 is 0%, and the influence degree of the maximum motion vector S8 is larger than the upper limit value (ψ + ξ) of the buffer section. , 100%.

  The mixing unit 22 mixes the maximum influence pixel S10 and the weighted average pixel S12 with the calculated mixing ratio S31 and outputs the result as the output pixel S13.

  Next, interpolation frame generation processing of the image processing apparatus will be described with reference to FIG.

  In this image processing apparatus, as shown in FIG. 21, first, a motion vector detection process for detecting a motion vector for one screen is performed (step S301). This motion vector detection process is performed in the same procedure as the procedure (step S201 to step S206) shown in the flowchart of FIG.

  Next, a vector extraction process is performed to extract a candidate vector S5 that is a motion vector S4 that may affect the pixel of interest from the motion vectors detected by this motion vector detection process (step S302). ). This vector extraction process is performed in the same procedure as the procedure (steps S207 to S211) shown in the flowchart of FIG.

  When the extraction of the candidate vector S5 is completed by the vector extraction process, the overlap resolution process is started.

  In the overlap resolution process, the mixture ratio calculation unit 21 extracts the candidate vector S5 having the maximum influence as the maximum motion vector S8 from the candidate vectors S5 held in the candidate vector holding unit 16 (Step S8). S303). Then, the mixture ratio calculation unit 21 calculates the mixture ratio S31 of the weighted average pixel S12 with respect to the maximum influence pixel S10 (step S304).

  Next, the influence degree normalization unit 18 normalizes the influence degree of the candidate vector S5 (step S305). By this normalization, a normalization candidate vector group S11 is generated and output to the weighted average processing unit 19. Then, the weighted average processing unit 19 performs weighted average processing of the reference pixel group S21 based on the influence degree of the normalization candidate vector group S11 output from the influence degree normalization unit 18 (step S306). The weighted average pixel S12 having the weighted average value obtained by the weighted average process as the pixel value is output to the mixing unit 10.

  Next, the maximum influence pixel S10 indicated by the maximum motion vector S8 is extracted from the current frame F [0] of the frame memory 1 by the selection unit 20 (step S307). Then, the pixel value of the weighted average pixel S12 output from the weighted average processing unit 19 and the pixel value of the maximum affected pixel S10 extracted from the current frame F [0] of the frame memory 1 are selected by the selection unit 10 from the above mixture ratio. Mixing is performed at S31 (step S308). By this mixing, the pixel value of the output pixel S13 is obtained, and the output pixel S13 is output.

  Next, it is determined whether or not the output of all the pixels on the interpolation frame F [−0.5] has been completed (step S309). If the output of all the pixels on the interpolation frame F [−0.5] has not been completed, the process returns to step S302, and the vector extraction process for the next pixel on the interpolation frame F [−0.5] is performed. Is started. On the other hand, when the output of all the pixels on the interpolation frame F [−0.5] is finished, this processing is finished.

  As described above, in the present embodiment, in the overlap resolution process, by calculating the mixture ratio S31, the continuous process is performed without switching the process based on the influence degree of the maximum motion vector S8 and the threshold ψ. Thus, an interpolation pixel can be output. As a result, a smoother image output can be obtained.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows an example of the some block cut out from the present flame | frame F [0]. It is a figure which shows an example of the block cut out from flame | frame F [-1] one frame before the present flame | frame F [0]. It is a figure which shows the motion vector S4 detected based on the offset of the partial image (block image of 8x8 pixels) S3a which most closely matches block S2. It is a figure which shows the interpolation motion vector S14 calculated | required based on the motion vector S4 calculated | required by the block matching part 2. FIG. It is a figure which shows the size of the image mapped on the interpolation frame F [-0.5] by the superimposition process part 4. FIG. It is a figure which shows the structure of the information field which stores the addition / maximum value mode, influence degree, and pixel value matched with every interpolation pixel. It is a figure which shows distribution of window function (omega). It is a figure which shows the characteristic showing the relationship between the motion vector of window function (omega), and influence degree. It is a figure which shows typically the relationship between the interpolation motion vector S14, the size Ia of the image to map, and influence S16. 4 is a flowchart showing a procedure of motion vector detection processing by a block matching unit 2 and interpolation vector output processing by a vector interpolation unit 3; It is a flowchart which shows the procedure of the superimposition process by the superimposition process part. It is a flowchart which shows the procedure of the normalization process by the normalization part. It is a figure which shows typically an example of the result of the interpolation frame production | generation process by the image processing apparatus of FIG. It is a block diagram which shows the structure of the image processing apparatus which concerns on the 2nd Embodiment of this invention. 16 is a flowchart showing a procedure of motion vector detection processing by the image processing apparatus of FIG. 15. It is a flowchart which shows the procedure of the vector extraction process by the image processing apparatus of FIG. FIG. 16 is a flowchart illustrating a procedure of duplication resolution processing by the image processing apparatus of FIG. 15. FIG. It is a figure which shows typically the selection method of the candidate vector by the motion vector selection part 13 of FIG. It is a block diagram which shows the structure of the image processing apparatus which concerns on the 3rd Embodiment of this invention. FIG. 21 is a flowchart illustrating a procedure of interpolation frame generation processing by the image processing apparatus of FIG. 20. FIG. It is a figure which shows typically the relationship between the mixture ratio S31 calculated by the mixture ratio calculation part 21 of FIG. 20, and the influence degree of maximum motion vector S8. It is a figure which shows typically the production | generation method of the conventional interpolation frame.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Frame memory 2 Block matching part 3 Vector interpolation part 4 Superposition process part 5 Window function table 6 Normalization part 12 Motion vector detection part 13 Motion vector selection part 14 Distance calculation part 15 Influence degree calculation part 16 Candidate vector holding part 17 Influence Degree determination unit 18 Influence normalization unit 19 Weighted average processing unit 20 Selection unit 21 Mixing ratio calculation unit 22 Mixing unit

Claims (6)

An image processing apparatus for performing frame interpolation for inserting an interpolation frame between a pair of temporally continuous frames,
Motion vector detection means for dividing one of the pair of frames into a plurality of blocks and detecting a motion vector between the pair of frames for each of the divided blocks;
For each block, an area that includes the block and is larger than the size of the block is cut out from one of the pair of frames, and the area cut out for each of the blocks is detected in the corresponding block. Arrangement means for arranging on the interpolation frame according to the motion vector;
A window function is applied to each of the clipped regions arranged on the interpolation frame, and the clipped material arranged on the interpolation frame is applied according to the value of the window function applied to each of the regions. and a determining means for determining a pixel value of the overlapping portion between the realm,
The determining means includes
When the maximum value of the window function values for each of the pixels in each region including the target pixel position of the overlapping portion is equal to or greater than a threshold value, the pixel of the pixel in the region corresponding to the maximum window function value The value is the pixel value at the target pixel position,
When all of the window function values for each of the pixels in each region are smaller than the threshold, the weighted average value of the pixel values of each pixel is calculated using the value of the window function for each of the pixels in each region, the image processing apparatus the calculated weighted average value issued, wherein to Rukoto the pixel value of the pixel of interest position. An image processing apparatus for performing frame interpolation for inserting an interpolation frame between a pair of temporally continuous frames,
Motion vector detection means for dividing one of the pair of frames into a plurality of blocks and detecting a motion vector between the pair of frames for each of the divided blocks;
For each block, an area that includes the block and is larger than the size of the block is cut out from one of the pair of frames, and the area cut out for each of the blocks is detected in the corresponding block. Arrangement means for arranging on the interpolation frame according to the motion vector;
A window function is applied to each of the clipped regions arranged on the interpolation frame, and the clipped material arranged on the interpolation frame is applied according to the value of the window function applied to each of the regions. Determining means for determining pixel values of overlapping portions between the determined regions;
With
The determining means includes
If the maximum value is larger than the upper limit of the preset predetermined range of the values of the window function for each pixel in each region encompassing the target pixel position of the overlapping portions between the disposed on the interpolation frame region , The pixel value of the pixel in the region corresponding to the value of the maximum window function as the pixel value of the target pixel position,
When all of the window function values for each of the pixels in each region are smaller than the lower limit value of the predetermined range, the weighted average value of the pixel values of the respective pixels using the window function value for each of the pixels in each region When the calculated weighted average value is the pixel value of the target pixel position, and the maximum value among the window function values for each of the pixels in each region is within the predetermined range, the maximum Weighting the pixel value of each pixel using the pixel value of the pixel corresponding to the maximum window function value and the value of the window function for each of the pixels of each region at a ratio according to the value of the window function images processor characterized by mixing the average value and the pixel value as the pixel value of the pixel of interest position.   The said determination means performs the normalization process according to the value of the window function with respect to each of the pixel of each area | region including the attention pixel position of the said overlapping part, The Claim 1 or Claim 2 characterized by the above-mentioned. Image processing device. An image processing method for performing frame interpolation in which an interpolation frame is inserted between a pair of temporally continuous frames,
A motion vector detection step of dividing one of the pair of frames into a plurality of blocks and detecting a motion vector between the pair of frames for each of the divided blocks;
An area that includes the block for each block and that is larger than the size of the block is cut out from one of the pair of frames, and the area cut out for each block is determined according to the detected motion vector of the corresponding block. A placement step for placing on the interpolation frame;
A window function is applied to each of the clipped regions arranged on the interpolation frame, and the clipped material arranged on the interpolation frame is applied according to the value of the window function applied to each of the regions. the a determination step of determining a pixel value of the overlapping portion between the realm
Equipped with a,
The determination step includes
When the maximum value of the window function values for each of the pixels in each region including the target pixel position of the overlapping portion is equal to or greater than a threshold value, the pixel of the pixel in the region corresponding to the maximum window function value The value is the pixel value at the target pixel position,
When all of the window function values for each of the pixels in each region are smaller than the threshold, the weighted average value of the pixel values of each pixel is calculated using the value of the window function for each of the pixels in each region, an image processing method the calculated weighted average value issued characterized to Rukoto the pixel value of the pixel of interest position. An image processing method for performing frame interpolation in which an interpolation frame is inserted between a pair of temporally continuous frames,
A motion vector detection step of dividing one of the pair of frames into a plurality of blocks and detecting a motion vector between the pair of frames for each of the divided blocks;
For each block, an area that includes the block and is larger than the size of the block is cut out from one of the pair of frames, and the area cut out for each of the blocks is detected in the corresponding block. An arrangement step of arranging on an interpolation frame according to a motion vector;
A window function is applied to each of the clipped regions arranged on the interpolation frame, and the clipped material arranged on the interpolation frame is applied according to the value of the window function applied to each of the regions. A determining step for determining pixel values of overlapping portions between the determined regions;
With
In the determination step ,
If the maximum value is larger than the upper limit of the preset predetermined range of the values of the window function for each pixel in each region encompassing the target pixel position of the overlapping portions between the disposed on the interpolation frame region , The pixel value of the pixel in the region corresponding to the value of the maximum window function as the pixel value of the target pixel position,
When all of the window function values for each of the pixels in each region are smaller than the lower limit value of the predetermined range, the weighted average value of the pixel values of the respective pixels using the window function value for each of the pixels in each region When the calculated weighted average value is the pixel value of the target pixel position, and the maximum value among the window function values for each of the pixels in each region is within the predetermined range, the maximum Weighting the pixel value of each pixel using the pixel value of the pixel corresponding to the maximum window function value and the value of the window function for each of the pixels of each region at a ratio according to the value of the window function mixing the average value, you characterized in that the pixel value and the pixel value of the target pixel position image processing method.   The normalization process according to the value of the window function with respect to each of the pixels of each region including the target pixel position of the overlapping portion is performed in the determination step. Image processing method.

Images