JP6792541B2 - Image generator, image generation method and image generation program - Google Patents

Description

The present invention relates to an image generator, an image generation method, and an image generation program.

A conventional camera controls motion blur and superimposed noise, which are image quality in the time direction, by changing the aperture time of the optical system. If the opening time of the camera is shortened, the motion blur is reduced, but the amount of received light is reduced by shortening the opening time, so that the superimposed noise increases. As described above, there is a trade-off relationship between the reduction of motion blur and the reduction of superimposed noise. That is, when it is desired to avoid noise superposition, it is necessary for a conventional camera to allow motion blur and secure a certain opening time. If you want to avoid motion blur, conventional cameras need to allow noise superposition and limit the aperture time.

Due to recent advances in semiconductor technology, the frame rate of moving images in high-speed cameras has greatly improved. Applications of high frame rate images acquired by high-speed cameras are classified into high image quality during image reproduction and high accuracy in image analysis.

The purpose of improving the image quality during image reproduction is to express smooth movement by approaching the upper limit of the frame rate that can be detected by the visual system (displayable on the display). Therefore, high image quality during image reproduction is premised on real-time reproduction (1x speed reproduction) on the display device.

On the other hand, the purpose of improving the accuracy of image analysis is to improve the accuracy of image analysis by using a high frame rate image that exceeds the visual detection limit. Image analysis by slow reproduction of high-speed moving objects such as athletes, FA / inspection, and automobiles is a typical application example.

The upper limit of the frame rate of the moving image input system and the upper limit of the frame rate of the moving image output system are asymmetric. At present, the upper limit of the frame rate of a high-speed camera, which is a moving image input system, exceeds 10,000 fps. On the other hand, the upper limit of the frame rate of the display device, which is a moving image output system, is 120 fps to 240 fps. Therefore, the moving image taken by the high-speed camera is used for slow reproduction (see Patent Document 1 and Non-Patent Document 1).

Japanese Unexamined Patent Publication No. 2004-201165

A. Golwelkar and J. Woods. Motion-compensated temporal filtering and motion vector coding using biorthogonal filter. IEEE Trans. Circuits Syst. Video, Vol. CSVT-17, No.4, 2007.

However, in the image generation process for the purpose of reducing motion blur and noise, it is premised that the image generation device samples frames at the reproduction frame rate. For this reason, conventional image generators do not sample frames with a time resolution higher than the playback frame rate.

A high frame rate image contains a group of frames sampled at high density in the time direction. When the image generator generates an image for real-time reproduction such as 30 Hz using a frame group sampled at a high density of 1000 Hz or the like, it is possible to generate an image for real-time reproduction with reduced motion blur. is there. However, the conventional image generator may not be able to generate an image for real-time reproduction in which motion blur and superimposed noise are reduced.

In view of the above circumstances, an object of the present invention is to provide an image generation device, an image generation method, and an image generation program capable of generating an image for real-time reproduction in which motion blur and superimposed noise are reduced. ..

One aspect of the present invention includes a moving image acquisition unit that acquires a target frame and a plurality of reference frames, and a first detection unit that detects a first corresponding position in the reference frame for each target pixel in the target frame. , Based on the first weight generation unit that generates the first weight coefficient of the pixel value for each first reference pixel in the vicinity of the first corresponding position, the pixel value of the first reference pixel, and the first weight coefficient. A first approximate value generation unit that generates a first approximate value of the pixel value of the target pixel, and a first update processing unit that updates the pixel value of the target pixel based on the first approximate value. It is an image generation device provided.

One aspect of the present invention is the above-mentioned image generation apparatus, in which a second detection unit that detects a second corresponding position of the reference frame for each target pixel of the updated pixel value, and a second corresponding position of the second corresponding position. An updated pixel value based on a second weight generation unit that generates a second weighting coefficient of a pixel value for each second reference pixel in the vicinity, a pixel value of the second reference pixel, and the second weighting coefficient. A second approximation value generation unit that generates a second approximation value of the pixel value of the target pixel, and a second that updates the pixel value of the target pixel of the updated pixel value again based on the second approximation value. It further includes an update processing unit.

One aspect of the present invention is an image generation method executed by an image generation device, in which a step of acquiring a target frame and a plurality of reference frames and a first step in the reference frame for each target pixel in the target frame. A step of detecting a corresponding position, a step of generating a first weighting coefficient of a pixel value for each first reference pixel in the vicinity of the first corresponding position, a pixel value of the first reference pixel and the first weighting coefficient. The image generation method includes a step of generating a first approximate value of the pixel value of the target pixel based on the above, and a step of updating the pixel value of the target pixel based on the first approximate value.

One aspect of the present invention includes a procedure for acquiring a target frame and a plurality of reference frames on a computer, a procedure for detecting a first corresponding position in the reference frame for each target pixel in the target frame, and the first aspect. Pixels of the target pixel based on a procedure for generating a first weighting coefficient of a pixel value for each first reference pixel in the vicinity of one corresponding position, a pixel value of the first reference pixel, and the first weighting coefficient. This is an image generation program for executing a procedure for generating a first approximate value of a value and a procedure for updating the pixel value of the target pixel based on the first approximate value.

According to the present invention, it is possible to generate an image for real-time reproduction in which motion blur and superimposed noise are reduced.

It is a figure which shows the example of the structure of the image generation apparatus in embodiment. It is a figure which shows the example of the structure of the initial composite image generation part in an embodiment. It is a figure which shows the example of the structure of the composite image generation part in embodiment. It is a figure which shows the example of the structure of the composite image update part in an embodiment. It is a flowchart which shows the example of the operation of the image generation apparatus in embodiment. It is a flowchart which shows the example of the operation of the initial composite image generation part in embodiment. It is a flowchart which shows the example of the detailed operation of the image generation apparatus in embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing an example of the configuration of the image generation device 1. The image generation device 1 is a device that generates an image. The image generation device 1 generates an image for real-time reproduction in which motion blur and superimposed noise are reduced. The image generation device 1 includes an input image storage unit 10, an initial composite image generation unit 20, a composite image storage unit 30, a composite image update unit 40, and an update end determination unit 50.

The input image storage unit 10 and the composite image storage unit 30 are non-volatile recording media (non-temporary recording media) such as a flash memory and an HDD (Hard Disk Drive). The input image storage unit 10 and the composite image storage unit 30 may have, for example, a volatile recording medium such as a RAM (Random Access Memory) or a register. The input image storage unit 10 stores an input image which is image data input to the image generation device 1.

A processor such as a CPU (Central Processing Unit) executes a program stored in the storage unit in a part or all of the initial composite image generation unit 20, the composite image update unit 40, and the update end determination unit 50. It is realized by. A part or all of each functional unit may be realized by using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

FIG. 2 is a diagram showing an example of the configuration of the initial composite image generation unit 20. The initial composite image generation unit 20 generates an initial composite image which is an initial image of the composite image to be updated. The initial composite image generation unit 20 includes an input image storage unit 21, a local neighborhood region setting unit 22, a corresponding position identification unit 23, a composite image generation unit 24, and a final pixel determination unit 25.

The input image storage unit 21 stores the target frame and the reference frame for the filtering process. The target frame and the reference frame are frames of a moving image having a high frame rate (high time resolution). When the upper limit of the frame rate of the display device which is the output system of the moving image is, for example, 240 fps, the high frame rate is a rate exceeding, for example, 240 fps.

The local neighborhood region setting unit 22 selects a reference frame from the reference frame candidates based on a predetermined order. The local neighborhood region setting unit 22 sets the pixels of the region near the local region (hereinafter referred to as “local neighborhood region”) of the pixel to be processed (hereinafter referred to as “target pixel”) in the target frame to the target frame. Hereinafter, the pixels in the region near the local area are referred to as "local neighborhood pixels".

The corresponding position identification unit 23 acquires the pixel value of the local neighborhood pixel from the local neighborhood region setting unit 22. The corresponding position identification unit 23 acquires the pixel value of the search range of the reference frame from the input image storage unit 21. The corresponding position identification unit 23 identifies the corresponding position in the reference frame for the target pixel in the target frame. The corresponding position identification unit 23 outputs the corresponding position information to the composite image generation unit 24.

FIG. 3 is a diagram showing an example of the configuration of the composite image generation unit 24. The composite image generation unit 24 generates a composite image based on the corresponding position and the reference frame. The composite image generation unit 24 outputs the pixel value (updated pixel value) of the composite image to the final pixel determination unit 25. The composite image generation unit 24 includes a reference frame setting unit 240, a reference pixel setting unit 241, a weight setting unit 242, an approximate value generation unit 243, a final reference frame determination unit 244, and a pixel value update processing unit 245. Be prepared.

The reference frame setting unit 240 acquires the reference frame of the input image from the input image storage unit 21. The reference pixel setting unit 241 sets a pixel at an integer pixel position near the corresponding position in the reference frame as a reference pixel.

The reference pixel setting unit 241 acquires the corresponding position information from the corresponding position identification unit 23. The reference pixel setting unit 241 acquires the pixel value of the local neighborhood region of the pixel at the corresponding position in all the predetermined reference frames from the reference frame setting unit 240.

The weight setting unit 242 determines the weighting coefficient of the reference pixel so that the approximation error is minimized when the pixel value in the local neighborhood region of the reference pixel is approximated by the pixel value in the local neighborhood region of the target pixel. The weight setting unit 242 outputs the weight coefficient of the reference pixel to the approximate value generation unit 243.

The approximation value generation unit 243 calculates the weighted sum of the reference pixel value and the weighting coefficient of the reference pixel as an approximation value of the target pixel value, which is the pixel value of the target pixel. The approximate value generation unit 243 outputs an approximate value of the target pixel value to the final reference frame determination unit 244.

The final reference frame determination unit 244 determines whether or not the process of calculating the approximate value of the target pixel value for all the predetermined reference frames has been completed. When the process of calculating the approximate value of the target pixel value is not completed for any of the reference frames, the composite image generation unit 24 repeats the process of calculating the approximate value of the target pixel value. When the processing for any of the pixels is not completed, the pixel value update processing unit 245 executes the processing for calculating the approximate value of the target pixel value for the next pixel.

When the process of calculating the approximate value of the target pixel value for all the predetermined reference frames is completed, the pixel value update processing unit 245 acquires the approximate value and the target pixel value of all the predetermined reference frames. To do. The pixel value update processing unit 245 calculates an approximate value and an average value of the target pixel values. The pixel value update processing unit 245 outputs the approximate value and the average value of the target pixel values to the final pixel determination unit 25 as the update value of the target pixel value. That is, the pixel value update processing unit 245 outputs the generated composite image (updated pixel value) to the final pixel determination unit 25 when the processing for all the predetermined pixels is completed.

Returning to FIG. 2, the description of the configuration of the initial composite image generation unit 20 will be continued. When the composite image is generated up to the final pixel of the composite image, the final pixel determination unit 25 outputs the generated composite image to the composite image storage unit 30.

FIG. 4 is a diagram showing an example of the configuration of the composite image updating unit 40. The composite image storage unit 30 stores the image generated by the initial composite image generation unit 20. The composite image update unit 40 updates the image generated by the initial composite image generation unit 20. The composite image update unit 40 includes a composite image storage unit 41, a local neighborhood region setting unit 42, a corresponding position identification unit 43, a composite image generation unit 44, and a final pixel determination unit 45.

The composite image storage unit 41 stores the composite image (generated image) generated by the initial composite image generation unit 20 and the reference frame group.
The local neighborhood region setting unit 42 selects a reference frame from the reference frame group based on a predetermined order. The local neighborhood region setting unit 42 sets the local neighborhood pixel of the target pixel in the target frame.

The corresponding position identification unit 43 acquires the pixel value of the local neighborhood pixel from the local neighborhood region setting unit 42. The corresponding position identification unit 43 acquires the pixel value of the search range of the reference frame from the composite image storage unit 41. The corresponding position identification unit 43 identifies the corresponding position in the reference frame for the target pixel in the target frame. The corresponding position identification unit 43 outputs the corresponding position information to the composite image generation unit 44.

The configuration of the composite image generation unit 44 includes the same configuration as that of the composite image generation unit 24. The composite image generation unit 44 operates in the same manner as the operation of the composite image generation unit 24. As a result of the operation, the composite image generation unit 44 generates an updated composite image based on the corresponding position and the reference frame.

When the generated composite image is updated to the last pixel of the composite image by the composite image generation unit 44, the final pixel determination unit 45 outputs the updated composite image to the update end determination unit 50.

Returning to FIG. 1, the description of the configuration of the image generation device 1 will be continued. The update end determination unit 50 is the composite image update unit 40 when the repetition end condition such that the difference squared sum of the composite image output from the composite image update unit 40 and the initial composite image is less than the threshold value is satisfied. Outputs the composite image output from.

Next, an example of the operation of the image generation device 1 will be described.
FIG. 5 is a flowchart showing an example of the operation of the image generation device 1. The image generation device 1 acquires image data to be filtered (step S1). The initial composite image generation unit 20 generates an initial composite image based on the acquired image data (step S2).

FIG. 6 is a flowchart showing an example of the operation of the initial composite image generation unit 20. The flowchart of FIG. 6 is the details of step S2 of FIG. The initial composite image generation unit 20 repeats the processes from steps S2a-2 to S2a-8 for all the target pixels in the frame of the acquired image data (step S2a-1).

The initial composite image generation unit 20 sets a local neighborhood pixel with respect to the target pixel (step S2a-2).
The initial composite image generation unit 20 repeats the processes from step S2a-4 to step S2a-7 for all the reference pixels in the reference frame (step S2a-3).

The initial composite image generation unit 20 acquires the pixel values of the local neighborhood pixels and the pixel values of the search range of the reference frame. The initial composite image generation unit 20 identifies the corresponding position in the reference frame and acquires the identified corresponding position information (step S2a-4).
The initial composite image generation unit 20 sets pixels (neighboring pixels) at integer pixel positions near the corresponding positions as reference pixels based on the corresponding position information (step S2a-5).

The initial composite image generation unit 20 calculates the weighting coefficient of the reference pixel based on the pixel value in the local neighborhood region of all the reference pixels and the pixel value in the local neighborhood region of the target pixel. The initial composite image generation unit 20 calculates the weighting coefficient of the reference pixel so that the approximation error is minimized when the pixel value in the local neighborhood region of the reference pixel is approximated by the pixel value in the local neighborhood region of the target pixel. (Step S2a-6).

The initial composite image generation unit 20 calculates the weighted sum of the reference pixel value and the weighting coefficient of the reference pixel as an approximate value of the target pixel value (step S2a-7).
The initial composite image generation unit 20 outputs the approximate values of all the reference frames of the acquired image data and the average values of the target pixel values to the composite image storage unit 30 as updated values of the target pixel values (step S2a-8). ).

FIG. 7 is a flowchart showing an example of the detailed operation of the image generation device 1. Steps S2-1 to S5 of the flowchart shown in FIG. 7 correspond to steps S2 to S5 of FIG.

The input image storage unit 21 stores a designated t-frame, which is a designated target frame, and a designated _rt number of reference frames. Each frame is sampled at the frame interval t (= jδ _t (j = 0, 1, ...)) Of the input signal of the time filter. f (x, y, t) (x = 0, ..., X-1, y = 0, ..., Y-1) is a pixel value at the position x, y of the t-th frame.

Local neighborhood region setting unit 22, a variable t representing the time position of the _frame, t ₌ t 0 -t b, _..., sets the value in the order of t 0 _-r t -b _t (step S2-1).

The initial composite image generation unit 20 executes the processes from step S2-2 to steps S2-4-4 for the frame at the set time position t. In other words, the initial composite image generating unit _20, to the (2b _{t +} r _t +1) sheets each frame, executes the process of step S2-4-4 from step S2-2.

Local neighborhood region setting unit 22, the target pixel f in the target frame (x, y, t) as a local neighborhood region _F b (x, y, t) _{for, (2b x +1) × (} 2b y +1) × ( 2b _t +1) consists of the pixel region {f (x-i, y -j, t-k) | i∈ [-b x, b x], j∈ [-b y, b y], k∈ [- _bt , _bt ]} is set.

Local neighborhood region setting unit 22, to the _{r t} Like each reference frame, executes the processing from step S2-3 to step S2-4-4. Local neighborhood region setting unit 22, reference frame to be processed the _{(t-r t} frame), 1, ..., set in the order of _{r t.} That is, the local neighborhood region setting unit 22 preferentially uses the reference frame close to the target frame for processing (step S2-2).

The corresponding position identification unit 23 identifies the corresponding position of the target pixel in the reference frame.
The details of steps S2-3 will be described.
The corresponding position identification unit 23 identifies the position in the reference frame that minimizes the sum of the squares of the differences between the local neighborhood region F _b (x, y) and the local neighborhood region of the corresponding point as shown in equation (1).

_{Here, e (v x, v y} , r t) is expressed by formula (2).

Equation (2) _{is, e (v x, v y} , r t -1) position of the corresponding point that minimizes _{(V (x, t'-r} t +1), V (y, t'-r t +1 )) Represents a search-centered process. r _t represents the time interval of up to reference frame. _(V x, _{v y)} represents the amount of displacement for the search center. The amount of displacement with respect to the target pixel position is expressed by the equation (3).

Here, round () is a function that returns the result digitized by rounding. The corresponding position identification unit 23 searches for a corresponding point with (1/4) pixel accuracy. The corresponding position identification unit 23 acquires the value of the decimal pixel position by bilinear interpolation.

Against a threshold theta _{e, e} (v _{x, v} y)> If the theta _e, the corresponding position identification unit 23 determines that the appropriate corresponding points do not _exist, setting a _(v x, _{v y)} do not do. The threshold value θ _e is set as an external parameter (step S2-3).

The composite image generation unit 24 uses pixels in the vicinity of the corresponding positions as reference pixels.
The details of step S2-4-1 (setting of the reference pixel) will be described.
In equation (3), and the pixel at the corresponding position in the _{(t-r t)} frame _{(x-V (x, t} -r t), y-V (y, t-r t)), in the vicinity Positioned pixels (x-V (x, tr _t ) + 1, y-V (y, tr _t )), (x-V (x, tr _t ), y-V (y, t) _{-rt) +1), (x-} V (x, t-r t) + 1, y-V (y, for _{t-r t)} +1) and consider each of the local neighborhood area.

Composite image generating unit 24, and centralizes the pixels included in each region near the raster scan order, calculating each vector of _{(2b x +1) × (2b} y +1) × (2b t +1) dimensional vector. _{(2b x +1) × (2b} y +1) × each vector of _(2b t +1) dimensional _{vector, R 0 (x, y,} t-r t) _{and, R 1 (x, y,} t-r t) When _is R 2 (x, y, t -r t) _{and, R 3 (x, y,} t-r t) and. For _example, elements of R 0 (x, y, t -r t) is expressed by the following equation (5).

Composite image generating unit _{24, R 0 (x, y,} t-r t) _{and, R 1 (x, y,} t-r t) _and, and R 2 (x, y, t -r t), R ₃ (x, y, tr _t ) is a weighted sum to approximate the local neighborhood region F _b (x, y, t) of the target pixel. The approximate error power E is expressed by the equation (6) (step S2-4-1).

The composite image generation unit 24 sets a weighting coefficient for the reference pixel.
The details of step S2-4-2 will be described.
Composite image generating unit 24 calculates an approximate error power E (x, y, t- r t) weighting factors _w 0 that _{minimizes, w 1, w 2, w} 3. The composite image generation unit 24 calculates the weighting coefficients w ₀ , w ₁ , and w ₂ based on the equation (7).

Here, for i, j = 0,1,2, the inner product value of the vector is expressed by the equation (8).

Here, t with a shoulder represents transposition. Composite image generating unit 24 uses the formula weight coefficients shown in _{(7) w 0, w 1} , w 2, and calculates the weight coefficient _{w 3} by the equation (9).

Equation (9) is the result of minimizing the approximate error power under the constraint that the sum of the weighting coefficients is 1. When no constraint is provided on the sum of the weighting coefficients, the weighting coefficients are expressed as in Eq. (10).

Here, for i, j = 0,1,2,3, the inner product value of the vector is expressed by the equation (11). The t on the shoulder represents transposition (step S2-4-2).

The composite image generation unit 24 generates an approximate value of the target pixel based on the reference pixel value and the weighting coefficient of the reference pixel.
The details of steps S2-4-3 will be described.
The composite image generation unit 24 calculates an approximate value of the target pixel value by multiplying the pixel value of each reference pixel by the weighting coefficient shown in the equation (11) as in the equation (12). The composite image generation unit 24 does not multiply the pixel value by the weight coefficient in the local neighborhood region unit, but multiplies the pixel value by the weight coefficient in the pixel unit. Composite image generating unit 24, _{r t} reference frames (the _{(t-r t) frame, (t = 1, ...,} r t)) of the reference pixel values for, is calculated as Equation (12).

_{Here, r i (x, y,} t-r t) , relative to i = 0, 1, 2, 3, is expressed by the equation (13). That is, equation (13) is a reference pixel values for the reference frame located at the time _{(t-r t)} (step S2-4-3).

The composite image generation unit 24 assigns reliability to the approximate value obtained from each reference frame. The composite image generation unit 24 generates the composite image ~ f (x, y, t) by updating the target pixel value with a weighted sum obtained by multiplying the approximate value by the reliability.

The details of step S2-4-4 (update of the target pixel value) will be described.
Composite image generating unit 24 on the basis of the reference pixel values for r _t reference frames represented by the equation (12), updates the target pixel value by the equation (14).

The first setting example of the coefficient _ak is expressed by the equations (15) and (16).

The second setting example of the coefficient _ak is expressed by the equations (17) and (18).

Here, the composite image generation unit 24 sets the number of samples satisfying | f (x, y, t) − ^ f (x, y, t, l _t ) | <ψ for each x, y, t. , R't (x, y, t). Incidentally, the range of the _{r 't (x, y,} t) is _{r t ≦ r't (x, y} , t) ≦ r t. The variable ψ is a threshold parameter given from the outside.

A third example of setting the coefficient _ak is expressed as in equations (19) and (20). Here, ε is a minute amount given to avoid division by zero (step S2-4-4).

The image generation device 1 identifies the corresponding points in the reference frame for each pixel of the composite image. Corresponding position identification unit _43, to the (2b _{t +} r _t +1) pieces of synthesized image _{f (x, y, t)} (t = t 0 -b t, ..., t 0 -r t -b t), The same process as that of step S2-3 of FIG. 7 is performed.

The corresponding position identification unit 43 uses ~ f (x, y, t ₀ ) as a target frame, and each composite image ~ f (x, y, t) (t = t _0-1 , ..., T) which is each reference frame. in ₀ -r _t) within to identify corresponding positions of the target pixel (step S3).

The image generation device 1 generates a composite image based on the image data acquired in step S1 and the corresponding points. The composite image generation unit 44 acquires the corresponding position information for each reference frame from the corresponding position identification unit 43. Composite image generating unit 44, the original frame _{f (x, y, t)} (t = t 0 -b t, ..., t 0 -r t -b t) acquired. Composite image generating unit 44, similar to the processing from step S2-4-1 to step S2-4-4, the composite image ~ f (x, y, t ) (t = t 0 -b t, ..., t ₀ −r _t − _bt ) is generated (step S4).

The image generation device 1 determines whether or not the composite image satisfies the iteration end condition. The end condition of the iteration is, for example, a condition that the sum of the difference squares between the composite image acquired in step S3 and the composite image output in step S4 is less than the threshold value. Acquiring update completion determination unit 50, the generated composite image _{~ f (x, y, t} ) (t = t 0 -b t, ..., t 0 -r t -b t) , and the combined image update section 40 To do. The update end determination unit 50 acquires the original frame from the input image storage unit 10. The update end determination unit 50 determines whether or not the end condition of the iterative process is satisfied (step S5).

When the composite image does not satisfy the repetition end condition (step S5: No). The image generation device 1 returns the process to step S3. When the composite image satisfies the repetition end condition (step S5: Yes), the update end determination unit 50 outputs the composite image ~ f (x, y, t ₀ ). That is, the update end determination unit 50 outputs an image of the position of the t-frame in which motion blur and superimposed noise are reduced (step S6).

As described above, the image generation device 1 of the embodiment includes the input image storage unit 10 (moving image acquisition unit), the corresponding position identification unit 23 (first detection unit), and the weight setting unit 242 (first weight generation unit). ), An approximate value generation unit 243 (first approximate value generation unit), and a pixel value update processing unit 245 (first update processing unit). The input image storage unit 10 acquires the target frame and a plurality of reference frames. The corresponding position identification unit 23 detects the first corresponding position in the reference frame for each target pixel in the target frame. The weight setting unit 242 generates a first weighting coefficient of the pixel value for each first reference pixel in the vicinity of the first corresponding position. The approximation unit 243 generates a first approximation value of the pixel value of the target pixel based on the pixel value of the first reference pixel and the first weighting coefficient. The pixel value update processing unit 245 updates the pixel value of the target pixel based on each first approximate value generated for the plurality of reference frames.

Thereby, the image generation device 1 of the embodiment can generate an image for real-time reproduction in which motion blur and superimposed noise are reduced. The image generation device 1 of the embodiment can generate a high-quality image for real-time reproduction (filtering process) from a high frame rate image that exceeds the visual detection limit.

The image generation device 1 of the embodiment further includes a corresponding position identification unit 43 (second detection unit). The image generation device 1 of the embodiment includes a second weight generation unit, a second approximate value generation unit, and a second update processing unit as each configuration of the composite image generation unit 44. The second detection unit detects the second corresponding position of the reference frame for each target pixel of the updated pixel value. The second weight generation unit generates a second weighting coefficient of the pixel value for each second reference pixel in the vicinity of the second corresponding position. The second approximate value generation unit generates a second approximate value of the pixel value of the target pixel of the updated pixel value based on the pixel value of the second reference pixel and the second weighting coefficient. The second update processing unit updates the pixel value of the target pixel of the updated pixel value again based on each second approximate value generated for the plurality of reference frames.

As a result, the image generation device 1 of the embodiment can generate an image for real-time reproduction with further reduced motion blur and superimposed noise.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

1 ... Image generator, 10 ... Input image storage unit, 20 ... Initial composite image generation unit, 21 ... Input image storage unit, 22 ... Local proximity area setting unit, 23 ... Corresponding position identification unit, 24 ... Composite image generation unit, 25 ... final pixel determination unit, 30 ... composite image storage unit, 40 ... composite image update unit, 41 ... composite image storage unit, 42 ... local neighborhood area setting unit, 43 ... corresponding position identification unit, 44 ... composite image generation unit, 45 ... Final pixel determination unit, 50 ... Update end determination unit, 240 ... Reference frame setting unit, 241 ... Reference pixel setting unit, 242 ... Weight setting unit, 243 ... Approximate value generation unit, 244 ... Final reference frame determination unit, 245 … Pixel value update processing unit

Claims (4)

A moving image acquisition unit that acquires a target frame and a plurality of reference frames,
A first detection unit that detects a first corresponding position in the reference frame for each target pixel in the target frame,
A first weight generation unit that generates a first weighting coefficient of a pixel value for each first reference pixel in the vicinity of the first corresponding position.
A first approximate value generation unit that generates a first approximate value of the pixel value of the target pixel based on the pixel value of the first reference pixel and the first weight coefficient.
A first update processing unit that updates the pixel value of the target pixel based on the first approximate value, and
An image generator comprising. A second detection unit that detects the second corresponding position of the reference frame for each target pixel of the updated pixel value, and
A second weight generation unit that generates a second weighting coefficient of a pixel value for each second reference pixel in the vicinity of the second corresponding position.
A second approximate value generation unit that generates a second approximate value of the pixel value of the target pixel of the updated pixel value based on the pixel value of the second reference pixel and the second weight coefficient.
A second update processing unit that updates the pixel value of the target pixel of the updated pixel value again based on the second approximate value, and
The image generator according to claim 1, further comprising. An image generation method executed by an image generation device.
Steps to get the target frame and multiple reference frames,
A step of detecting a first corresponding position in the reference frame for each target pixel in the target frame, and
A step of generating a first weighting coefficient of a pixel value for each first reference pixel in the vicinity of the first corresponding position, and
A step of generating a first approximate value of the pixel value of the target pixel based on the pixel value of the first reference pixel and the first weighting coefficient, and
An image generation method including a step of updating the pixel value of the target pixel based on the first approximate value. On the computer
The procedure to get the target frame and multiple reference frames,
A procedure for detecting a first corresponding position in the reference frame for each target pixel in the target frame, and
A procedure for generating a first weighting coefficient of a pixel value for each first reference pixel in the vicinity of the first corresponding position, and
A procedure for generating a first approximate value of the pixel value of the target pixel based on the pixel value of the first reference pixel and the first weighting coefficient, and
An image generation program for executing a procedure for updating the pixel value of the target pixel based on the first approximate value.

Images