JP6652290B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

The present invention relates to an image processing apparatus, image processing method, and a program, it is suitably used to reduce the free Murrell noise in the image.

  In the video signal obtained from the image pickup system, noise caused by the image pickup element, noise caused by circuit elements around the image pickup element, and the like are superimposed. Noise superimposed on a video signal obtained from such an imaging system is roughly classified into two types of noise: fixed pattern noise and random noise. The former is attributable to variations in pixel amplifiers due to pixel defects in the image sensor. The latter is caused by dark current and shot noise due to the image sensor, noise generated from circuit elements, and the like.

  As a method for removing or reducing noise superimposed on the video signal as described above, a noise removal process such as a process using a one-dimensional filter or a two-dimensional filter can be considered. As a process using a one-dimensional filter or a two-dimensional filter, there is a filter process using a pixel value of a pixel adjacent to a target pixel. In the following description, “noise removal” is referred to as “NR” as necessary (NR is an abbreviation for Noise Reduction).

  In addition, cyclic NR processing is used as NR processing of a video signal of a moving image, in particular, NR processing of random noise that changes variously with time. In the cyclic NR process, a difference signal between a current video signal and a video signal of a past frame is obtained, and a signal obtained by multiplying the difference signal by a cyclic coefficient is subtracted from the current video signal to remove random noise. .

In this cyclic NR processing, a signal obtained by multiplying the above-described difference signal by a cyclic coefficient is used as a noise signal to be subtracted from the current video signal. Therefore, if the cyclic coefficient is set to a large value, the effect of the NR also increases. On the other hand, the difference signal includes not only the noise component but also the motion component of the edge of the subject. Therefore, when the cyclic NR processing is performed without any contrivance, an afterimage occurs. Therefore, there is a problem of how to accurately separate the noise component and the motion component, that is, how to accurately detect the motion component, determine the noise component and the motion component satisfactorily, and determine the cyclic coefficient only with the noise component. .

  Thus, Patent Document 1 discloses a cyclic noise reduction device provided with a motion detection circuit. This motion detection circuit includes a smoothing circuit, a difference circuit, and a motion determination circuit. The smoothing circuit is a circuit for reducing high frequency noise of a video signal. The difference circuit is a circuit for calculating a difference between an output signal from the smoothing circuit and an output signal from the frame memory. The motion determination circuit is a circuit for determining a motion by comparing an output signal of the difference circuit with a preset signal level (threshold). Based on the result of the determination by the motion determination circuit, the value of the cyclic coefficient is determined, and the cyclic NR process is performed.

JP-A-6-225178

  However, the technique described in Patent Literature 1 requires a processing circuit for judging motion, and has a problem that the cost is increased by adding the processing circuit. Further, in the technique described in Patent Document 1, it is necessary to accurately set a threshold value for determining a motion. However, a threshold value that does not cause an afterimage for any subject (for example, a fast-moving subject). There is also a problem that setting is not easy.

  Further, in the technique described in Patent Literature 1, an output signal of a frame memory based on a current frame and a frame immediately before the current frame is used as a motion determination video signal. Since the motion is determined based on the difference between these frames, it is impossible to detect the motion when the subject moves during the time of one frame, and an afterimage still occurs for the fast-moving subject. There's a problem.

  Further, in the technique described in Patent Literature 1, a motion is determined based on a difference between frames. Therefore, it takes at least two frames of processing time to acquire an image of two frames. Then, after obtaining the video signals of these two frames, a subtraction process is performed to determine the motion by calculating the difference between the video signals of these two frames, and a cyclic coefficient is derived from the result of the motion determination to obtain the current frame. On the other hand, noise reduction processing is performed. For this reason, it is possible to obtain an NR-processed image from at least the fourth frame. These processing times affect the performance of the image processing device. Thus, the technique described in Patent Document 1 has a problem that it is not easy to shorten the processing time.

  The present invention has been made in view of the above problems, and an object of the present invention is to easily and reliably suppress occurrence of an afterimage in a captured image.

An image processing apparatus according to an aspect of the present invention is an image processing apparatus that processes an image captured by an imaging unit capable of setting an imaging condition for each first direction of an image, the current frame being captured by the imaging unit. Using the pixel signal of the first pixel of the image of FIG. 1, the pixel signal of the second pixel , and the pixel signal of the first pixel of the image of the frame earlier than the current frame , It includes a processing means for performing processing to reduce noise, wherein the first pixel second pixel is located in said first direction and perpendicular direction is different from pixel to each other, the first And the pixel signal of the second pixel is a pixel signal obtained under different imaging conditions.

  According to the present invention, it is possible to easily and reliably suppress occurrence of an afterimage in a captured image.

FIG. 2 is a diagram illustrating a configuration of a digital camera. FIG. 3 is a diagram illustrating a first example of a configuration of an NR processing unit. FIG. 4 is a diagram illustrating an arrangement of pixels on a vertical line. FIG. 3 is a diagram illustrating a video signal output from a first pixel. FIG. 9 is a diagram illustrating a video signal output from a second pixel. FIG. 11 is a diagram illustrating a second example of the configuration of the NR processing unit. It is a figure showing a pixel on a vertical line. FIG. 14 is a diagram illustrating a third example of the configuration of the NR processing unit. FIG. 4 is a diagram illustrating a video signal when a subject is stationary or moving. It is a figure showing the parameter of NR processing. FIG. 14 is a diagram illustrating a fourth example of the configuration of the NR processing unit.

Hereinafter, embodiments will be described with reference to the drawings.
(First Embodiment)
First, a first embodiment will be described.
FIG. 1 is a block diagram illustrating an example of a functional configuration of a digital camera as an example of an imaging device. In the present embodiment, the digital camera 100 has a function of capturing a still image and a moving image of a subject.
The operation unit 101 is a user interface for a user to input and set various commands to the digital camera 100. For example, the operation unit 101 has, as an input device, mechanical switches and buttons having a function of setting various commands. Alternatively, the operation unit 101 may be configured by forming buttons having the same functions as those switches and buttons on a display device such as a touch panel type liquid crystal display and displaying the same. The operation unit 101 is used, for example, when turning on / off a power supply, setting / changing / checking shooting conditions, and checking a shot image.

  The operation unit 101 also includes a shutter switch. When the shutter switch is half-pressed, it functions as the first shutter switch SW1, and when the shutter switch is fully pressed, it functions as the second shutter switch SW2.

  By notifying that the shutter switch is functioning as the first shutter switch SW1, the system control unit 102 is instructed to start AF processing, AE processing, and the like, which will be described later. On the other hand, the notification that the shutter switch is functioning as the second shutter switch SW2 instructs the system control unit 102 to start a series of shooting processing operations. A series of photographing processing operations includes reading of an image signal from the image sensor 103, A / D conversion of the read image signal, image processing of the A / D converted image signal, and conversion of the image processed image signal into a recording format. , And processing of recording image data in the image recording unit 112.

  The system control unit 102 controls the operation of each unit of the digital camera 100 according to an instruction from the operation unit 101. The system control unit 102 includes a CPU, a ROM for storing programs executed by the CPU, a RAM as a storage area for reading programs and a storage area for a work area, and the like. The system control unit 102 calculates the luminance level of the subject from the digital image data output from the image processing unit 105, and automatically determines at least one of the shutter speed (exposure time) and the aperture according to the shooting mode. Is performed, an automatic exposure control (AE) process is performed.

  In the digital camera 100 of the present embodiment, the exposure mechanism 108a has a function of an aperture. On the other hand, the shutter, which is one of the factors that determine the exposure, is an electronic shutter. The control by the system control unit 102 is executed so that the image signal is read from the image sensor 103 at a predetermined time (shutter speed (exposure time)).

  Upon receiving the notification of the result of the AE process from the system control unit 102, the mechanical drive unit 108 operates the exposure mechanism 108a with the aperture and shutter speed under the control of the electronic shutter. By doing so, the optical path and the light amount between the lens 107a and the image sensor 103 can be secured. Thus, it becomes possible to expose the subject to the image sensor 103 under the exposure conditions determined by the AE processing.

  Further, the system control unit 102 drives the focus lens of the lens optical system 107a using the lens driving unit 107, detects a change in the contrast of the digital image data output from the image processing unit 105, and automatically adjusts the contrast based on the change. A focus control (AF) process is performed.

  The operation unit 101 of the present embodiment is also provided with a zoom lever (not shown) for performing a zoom function. As a result, a signal instructing the movement of the zoom lens of the lens optical system 107a to the zoom position linked with the zoom lever is notified to the system control unit 102. Based on this signal, the system control unit 102 uses the lens driving unit 107 to move the zoom lens of the lens optical system 107a to a desired zoom position. By controlling the above-described lens optical system 107a, it is possible to realize photographing with a lens arrangement that provides a desired zoom position.

  Further, the system control unit 102 notifies the A / D conversion unit 104 of the gain adjustment amount according to the ISO sensitivity. The ISO sensitivity may be a fixed ISO sensitivity set by the user or an ISO sensitivity dynamically set by the system control unit 102 based on the result of the AE process. The setting of the ISO sensitivity will be described later.

  Further, the system control unit 102 determines whether or not the light emission operation of the flash unit 110 at the time of the main shooting is necessary according to the flash setting, the shutter speed (exposure time) based on the result of the AE processing, and the like. When the flash emission is determined, the system control unit 102 instructs the EF processing unit 109 to execute the flash emission. When receiving an instruction to execute flash emission from the system control unit 102, the EF processing unit 109 controls the flash unit 110 and causes the flash unit 110 to emit light at the timing when the shutter of the exposure mechanism 108a opens.

In the present embodiment, the image sensor 103 has a stacked CMOS sensor, which is a type of photoelectric conversion device. The image sensor 103 converts the subject optical image formed via the lens 107a and the exposure mechanism 108a into an analog electric signal (analog image data) for each pixel.
In the present embodiment, as an example of an imaging condition, a function of a stacked CMOS sensor that can change a shutter speed (exposure time) and an ISO sensitivity gain adjustment amount (setting of a gain for a pixel signal) for each horizontal line of the image sensor 103 is described. Used. Thus, in the present embodiment, the direction of the horizontal line is the first direction.

  As described above, in the stacked CMOS sensor, the shutter speed (exposure time) related to the exposure and the ISO sensitivity gain adjustment amount can be changed in the lines adjacent to each other. In the present embodiment, the stacked CMOS sensor has a configuration in which the same set value can be set for every other line as the shutter speed (exposure time) and the ISO sensitivity gain adjustment amount, and imaging can be performed. By using the above functions, it is possible to realize a cyclic NR process of the present embodiment described later.

  In the present embodiment, a case where the image sensor 103 includes a stacked CMOS sensor will be described as an example. However, the image sensor 103 is not necessarily limited to a device having a stacked CMOS sensor. The imaging element 103 may include a single-layer CCD sensor, CMOS sensor, or the like that is generally used. That is, as long as the sensor has a function of changing the exposure time and the ISO sensitivity gain adjustment amount for each line of the image sensor 103, the sensor included in the image sensor 103 may be a single-layer sensor.

  The A / D converter 104 generates and outputs digital image data by performing correlated double sampling, gain adjustment, A / D conversion, and the like on the analog image data output from the image sensor 103. . The gain (amplification rate) is set by being notified from the system control unit 102 to the A / D conversion unit 104. When a large gain is set, the signal level increases, but on the other hand, the noise component included in the image also increases. The gain setting here is changed according to the ISO sensitivity setting. Details of the setting of the gain will be described later.

  The image processing unit 105 performs image processing on the digital image data output from the A / D conversion unit 104. The image processing unit 105 performs image processing such as, for example, white balance correction, edge enhancement processing, NR processing, pixel interpolation processing, gamma correction processing, and color difference signal generation. As image-processed digital image data, for example, a YUV image Processing such as outputting data is performed.

The EVF display unit 106 includes a display device such as an LCD (Liquid Crystal Display), and displays an image (not shown) that has been subjected to D / A conversion processing on digital image data on which image processing has been performed by the image processing unit 105.
The format conversion unit 111 generates a recording data file based on, for example, DCF (Design rule for Camera File System) for the digital image data output from the image processing unit 105. In the process of generating a data file, the format conversion unit 111 performs encoding into a JPEG format or a Motion JPEG format, generates a file header, and the like.

The image recording unit 112 records the data file for recording generated by the format conversion unit 111 in a built-in memory of the digital camera 100, a removable medium mounted on the digital camera 100, or the like.
The external connection unit 113 is an interface for connecting the digital camera 100 to an external device such as a PC (personal computer) or a printer. The external connection unit 113 communicates with an external device in accordance with general standards such as USB (Universal Serial Bus) and IEEE 1394 to exchange image data and the like, and utilizes the functions of each external device. .

Next, an example of the operation of the digital camera 100 of the present embodiment will be described.
When the user turns on a power switch (not shown), which is one of the operation units 101, the system control unit 102 detects this and sends a battery or AC input (not shown) to each component of the digital camera 100. Supply power.
When power is supplied, the digital camera 100 according to the present embodiment starts a display operation using an EVF (Electronic View Finder). First, when power is supplied, the mechanical shutter provided in the exposure mechanism 108a opens, and the image sensor 103 is exposed. The charge accumulated in each pixel of the image sensor 103 is sequentially read at a predetermined frame rate and output to the A / D converter 104 as analog image data. As described above, in the present embodiment, the electric charge accumulated in each pixel of the image sensor 103 is sequentially read out at a predetermined frame rate. Get an image for

As described above, the A / D converter 104 generates digital image data by performing correlated double sampling, gain adjustment, A / D conversion, and the like on the analog image data output from the image sensor 103. And output. Here, the gain adjustment will be described.
In the image sensor 103, the output signal level of the analog electric signal changes depending on the exposure amount. Since the exposure amount increases for a bright subject, the output signal level of the analog electric signal also becomes large. On the other hand, since the exposure amount decreases for a dark subject, the output signal level of the analog electric signal also becomes small. When an analog electric signal having the above-described level fluctuation is input to the A / D converter 104 and output without adjusting the gain in the A / D converter 104, the output digital electric signal also fluctuates in level. .

  In contrast, a digital camera generally has a gain that keeps the output signal level of the digital electric signal from the A / D converter 104 constant regardless of the brightness of the subject (output signal level of the analog electric signal). Is set according to the brightness of the subject.

The above gain changes and is adjusted according to the setting of the ISO sensitivity which is one of the imaging conditions. In other words, a higher value is set as the gain in the A / D converter 104 when the subject is dark and at high ISO sensitivity than when the subject is bright and at low ISO sensitivity. For this reason, the noise component included in the image is larger at the time of the high ISO sensitivity than at the time of the low ISO sensitivity due to the amplification effect by the higher gain.
As described above, the ISO sensitivity related to the setting of the gain of the A / D conversion unit 104 as described above may be a fixed ISO sensitivity set by the user, or the system control unit 102 may set the result of the AE processing. ISO sensitivity may be dynamically set based on the ISO sensitivity.

  As described above, the image processing unit 105 performs image processing on the digital image data output from the A / D conversion unit 104, and outputs, for example, YUV image data as image-processed digital image data. The image processing includes, for example, white balance correction, edge enhancement processing, NR processing, pixel interpolation processing, gamma correction processing, and color difference signal generation.

Further, the EVF display unit 106 sequentially displays the images that have been subjected to the D / A conversion processing (not shown) using the digital image data output from the image processing unit 105.
The system control unit 102 repeats the above-described execution of the display processing by the EVF unless the notification that the shutter switch is functioning as the first shutter switch SW1 (the half-press notification) is received from the operation unit 101.

  Upon receiving a notification from the operation unit 101 that the shutter switch is functioning as the first shutter switch SW1, the system control unit 102 performs the AF process and the AE process using the latest captured image at that time. , The focus position and the exposure condition are determined. At the same time, the system control unit 102 also determines whether or not the flash unit 110 needs to emit light (performs EF processing). Whether or not the flash unit 110 emits light is determined automatically by detecting the darkness around the digital camera 100, regardless of whether it is set in advance using the operation unit 101 or set in response to the result of the AE process. May be set.

  After that, the system control unit 102 continues to wait without performing a shooting operation until receiving a notification (a full-press notification) that the shutter switch is functioning as the second shutter switch SW2. If the notification that the shutter switch is functioning as the first shutter switch SW1 is interrupted before the notification that the shutter switch is functioning as the second shutter switch SW2 is interrupted, the system control unit 102 displays the image using the EVF. Resume processing.

  Upon receiving notification that the shutter switch is functioning as the second shutter switch SW2, the system control unit 102 controls the EF processing unit 109 when the flash unit 110 emits light, and causes the flash unit 110 to perform pre-emission. Let it do. Then, the system control unit 102 performs processing such as calculation of the main light emission amount and weighting of the EF frame based on the result of the pre-light emission. Then, the system control unit 102 instructs the EF processing unit 109 to emit light from the flash unit 110 at the calculated main light emission amount, and shifts to the main photographing processing. When the flash unit 110 does not emit light, the system control unit 102 shifts to the main photographing process without performing the above-described light control.

  In the main photographing process after receiving the notification that the shutter switch is functioning as the second shutter switch SW2, on the image sensor 103 under the above-described AF process, AE process, and in some cases, EF process. To expose and form an object optical image. After that, the analog electric signal in pixel units from the image sensor 103 is converted into digital image data by the A / D converter 104, and image processing is performed by the image processor 105 using the digital image data. Then, the digital image data that has been subjected to the image processing is converted into a data file format for recording by the format conversion unit 111, and is recorded on the recording medium by the image recording unit 112.

  The above is an example of the operation of the digital camera 100 at the time of shooting a still image. The operation is performed when the still image shooting mode is selected on the operation unit 101 with a selector switch (not shown). On the other hand, when the selector switch is set to the moving image shooting mode, the operation mainly when the shutter switch functions as the first shutter switch SW1 and the second shutter switch SW2 is the still image shooting mode. different. Hereinafter, an example of the operation of the digital camera 100 when shooting a moving image will be described.

  Upon receiving the notification that the shutter switch is functioning as the first shutter switch SW1 in the moving image shooting mode, the system control unit 102 performs the AF process and the AE process as in the still image shooting mode. However, EF processing is not normally performed. Further, during the moving image shooting mode, the AF process and the AE process are performed at certain intervals, and as for the display by the EVF, the image is continuously updated and displayed at a predetermined frame rate unless the moving image shooting mode is switched.

  In the moving image shooting mode, when the notification that the shutter switch is functioning as the second shutter switch SW2 is received, the main shooting process is executed. Among the main photographing processes, the AF process and the EVF process are the same as when the shutter switch functions as the first shutter switch SW1. On the other hand, AE processing and NR processing for a moving image, which will be described later, are executed as the AE processing and the NR processing in the image processing unit 105. That is, the method of the AE processing and the method of the NR processing are different between the case of the still image shooting mode and the case of the moving image shooting mode.

  Further, the format conversion unit 111 converts the digital image data subjected to the image processing into a moving image format (for example, a Motion JPEG format) as a recording data file format, and records the data by the image recording unit 112. Record on the medium.

The above is an example of the configuration and operation of the digital camera 100 of the present embodiment. Hereinafter, an example of the image processing method according to the present embodiment, particularly, an example of the NR processing used when capturing a moving image will be described.
FIG. 2 is a block diagram illustrating an example of a configuration of the NR processing unit 200 in the image processing unit 105.

  A subtractor 201 subtracts the video signal Mem_in1 of a previous frame from the current frame (hereinafter, referred to as a “current frame”) of the current frame to be processed to obtain a difference signal. In the present embodiment, the first difference signal deriving process for deriving the first difference signal in this manner is executed. A value obtained by multiplying the difference signal by the cyclic coefficient K by the multiplier 202 is used as a noise component, and the noise component is subtracted by the subtractor 205 from the video signal of the current frame Cur_in1, thereby realizing noise removal. In the present embodiment, the fourth difference signal deriving process for deriving the fourth difference signal in this manner is performed.

  However, the difference signal between the video signal Cur_in1 of the current frame and the video signal Mem_in1 of the previous and previous frames may include a change in signal level due to movement of the subject, in addition to the noise component. Therefore, in the above-described processing, the difference due to the fluctuation of the signal level is erroneously recognized as noise, and unnecessary NR processing is performed. As a result, adverse effects such as an afterimage being included in the image appear.

  Therefore, a signal output from a second pixel vertically adjacent to the first pixel that outputs the video signal Cur_in1 of the current frame and the video signal Mem_in1 of the previous and previous frames is used. Specifically, in the present embodiment, the video signal Cur_in2 of the current frame and the video signal output Mem_in2 of the previous frame before that output from the second pixel are used. The frame image of the past frame by the video signal Mem_in2 is, for example, a frame image one frame before the current frame. In the present embodiment, by doing so, the above-described adverse effects are avoided. The details will be described below.

  In FIG. 2, the video signal Cur_in1 of the current frame and the video signal Mem_in1 of the previous and previous frames are outputs from the same one pixel (first pixel) on a predetermined horizontal line. The video signal Cur_in2 of the current frame and the video signal Mem_in2 of the previous frame before the current frame are outputs from the same one pixel (second pixel) vertically adjacent to the first pixel.

FIG. 3 is a diagram schematically illustrating an example of an array of pixels on a vertical line.
In the present embodiment, a total of four pixels including one R pixel, one B pixel, and two G pixels shown in FIG. Here, the R pixel is a pixel provided with a red filter, the B pixel is a pixel provided with a blue filter, and the G pixel is a pixel provided with a green filter.

  However, the arrangement of the pixels on the vertical line is not limited to this. For example, as shown in FIG. 3B, one R pixel and one G pixel, or one B pixel and one G pixel may be treated as a unit of one pixel described above. However, in this case, since the light receiving sensitivity of the sensor (image sensor 103) generally has wavelength dependence, the following points need to be noted. That is, the gain in the unit of one pixel including the R pixel and the gain in the unit of one pixel including the B pixel are set in consideration of the wavelength dependency due to the difference between the R pixel and the B pixel. Is required.

The first pixel and the second pixel in a unit of one pixel as described above are pixels on a horizontal line adjacent to each other in the vertical direction. The shutter speed (exposure time) and the ISO sensitivity gain adjustment amount can be changed between the first pixel and the second pixel by the function of the stacked CMOS sensor.
Assuming that the exposure time and gain for the first pixel are Et1 and G1, respectively, and that the exposure time and gain for the second pixel are Et2 and G2, respectively, in this embodiment, the following equations (1) and (2) It has the relationship of the formula.
Et1 <Et2 ≦ (time required to generate one frame of moving image) (1)
G1> G2 (2)

The exposure times Et1 and Et2 and the gains G1 and G2 are read out from the ROM or RAM in the digital camera 100 and set by the AE process performed by the system control unit 102 at the time of capturing a moving image.
The above-described exposure times Et1 and Et2 and gains G1 and G2 are determined by the following equation (3) when the first pixel and the second pixel receive the same amount of light Y (= Y0). It is decided to be.
Et1 × G1 × Y0 = Et2 × G2 × Y0 (3)

Therefore, if the amount of reflected light received from the subject is the same for the first pixel and the second pixel, when sampling is performed in the time required to generate one frame of the moving image, the first pixel and the second pixel The output is read as the same level.
FIGS. 4 and 5 show a difference between the video signal Cur_in1 of the current frame and the video signal Cur_in1 and Mem_in1 when the edge of the subject image moves between the video signal Mem_in1 of the previous frame and the previous frame. It is a figure which shows an example of a signal notionally. FIG. 4 shows a video signal output from the first pixel, and FIG. 5 shows a video signal output from the second pixel.

  As shown in FIG. 4, it is assumed that the edge portions 401 and 402 of the subject image have moved between the video signal Cur_in1 of the current frame and the video signal Mem_in1 of the previous frame before that. As a result, it is assumed that a level change has occurred in the output signals 403 and 404 between the video signal Cur_in1 of the current frame of the first pixel and the video signal Mem_in1 of the previous frame of the first pixel before that. In this case, a difference signal Δ1 between the two video signals Cur_in1 and Mem_in1 is obtained.

Here, although details will be described later, in order to extract a noise component superimposed on the video signal Cur_in1 of the current frame, it is preferable to use a video signal from which the noises 405a and 405b have been removed as much as possible as the video signal Mem_in1 of the current frame. .
On the other hand, even between the video signal Cur_in2 of the current frame of the second pixel vertically adjacent to the first pixel and the video signal Mem_in2 of the previous frame before that, the edge portions 501 and 502 of the subject image are also generated. The movement causes a level change in the output signals 503 and 504. In this case, a difference signal Δ2 between the two video signals Cur_in2 and Mem_in2 is obtained.

  4 and 5, the difference between the difference signals Δ1 and Δ2 is a difference in the amount of noise due to a difference in the setting of the ISO sensitivity gain. On the other hand, the level variation of the output signals 403, 404, 503, 504 due to the movement of the edge portions 401, 402, 501, 502 of the subject image becomes the same signal.

  Therefore, when the difference between the difference signals Δ1 and Δ2 is obtained, the level fluctuations of the output signals 403, 404, 503 and 504 due to the movement of the edge portions 401, 402, 501 and 502 of the subject image are canceled out. Therefore, the difference of only the noise component corresponding to the setting difference of the ISO sensitivity gain is extracted as the difference output signal Δ12.

  Therefore, in the present embodiment, as shown in FIG. 1, the subtractor 201 subtracts the video signal Mem_in1 of the previous frame from the video signal Cur_in1 of the current frame with respect to the first pixel, thereby obtaining a difference signal Δ1 Ask for. In addition, for the second pixel, the subtractor 203 subtracts the video signal Mem_in2 of the previous frame from the video signal Cur_in2 of the current frame to obtain a difference signal Δ2. In the present embodiment, the second difference signal deriving process for deriving the second difference signal in this manner is executed.

  Then, a difference signal Δ12 is obtained by subtracting the difference signal Δ2 from the difference signal Δ1 by the subtractor 204. In the present embodiment, the third difference signal deriving process for deriving the third difference signal in this manner is executed. The difference signal Δ12 is multiplied by a cyclic coefficient K by a multiplier 202. In the present embodiment, the fourth signal derivation process is executed in this manner. A value obtained by multiplying the difference signal Δ12 by the cyclic coefficient K is subtracted from the video signal Cur_in1 of the current frame of the first pixel by the subtractor 205 to obtain an output signal of the NR processing unit 200. By doing so, it is possible to reliably remove only the noise component present in the video signal Cur_in1 of the current frame of the first pixel without being affected by the level change of the output signal due to the movement of the subject.

The video signal Cur_in1 of the current frame of the first pixel from which noise has been removed as described above is used as the video signal Mem_in1 of the past frame after the next frame. As described above, it is possible to obtain a moving image in which afterimages are reduced and noise is reliably reduced without adding a motion detection circuit.
Note that, similarly to the video signal Cur_in1 of the current frame of the first pixel, the video signal Cur_in2 of the current frame of the second pixel of the second pixel is also used as the video signal Mem_in2 of the past frame after the next frame.

  In addition, in the above-described cyclic NR processing, when a moving image of the first frame (first frame) is captured, the video signal Mem_in1 of the past frame does not normally exist. Therefore, it is necessary to prepare the video signal Mem_in1 of the past frame in advance.

On the other hand, it is preferable to use a video signal from which noise has been removed as much as possible as the video signal Mem_in1 of the past frame because the image quality can be improved. Therefore, for example, the following video signal can be used as the video signal Mem_in1 of the past frame.
The video signal of the frame immediately before the video signal Cur_in1 of the current frame or the video signal obtained by performing another NR process on the video signal Cur_in1 of the current frame can be used as the video signal Mem_in1 of the past frame.
As another NR process, for example, a filter process can be performed on a target pixel using pixel values of peripheral pixels of the target pixel provided for a still image. Alternatively, a low ISO sensitivity may be set so that the gain is lower than that of the video signal Cur_in1 of the current frame, and the video signal Cur_in2 having a small amount of noise may be used as the video signal Mem_in1 of the past frame.

  As described above, in the present embodiment, for the first pixel and the second pixel, the frame images based on the video signals Mem_in1 and Mem_in2 of the previous and previous frames are subtracted from the video signals Cur_in1 and Cur_in2 of the current frame to obtain a difference signal. Find Δ1 and Δ2. Then, the difference signal Δ2 is subtracted from the difference signal Δ1 to obtain the difference signal Δ12, and the difference signal Δ12 is multiplied by the cyclic coefficient K, and this is subtracted from the video signal Cur_in1 of the current frame of the first pixel. Therefore, the noise component and the motion component can be accurately separated without adding a motion detection circuit, and the NR process can be performed with the edge of the subject image maintained. Therefore, it is possible to suppress noise and suppress the occurrence of an afterimage at low cost.

(Second embodiment)
Next, a second embodiment will be described. In the first embodiment, the video signal of the second pixel adjacent to the first pixel in one vertical direction has been described as an example. On the other hand, in the present embodiment, one of the video signal of the second pixel adjacent to the first pixel in one direction in the vertical direction and the video signal of the third pixel adjacent to the first pixel in the other direction is used. Choose As described above, the present embodiment and the first embodiment mainly differ in the configuration and processing by using the video signal of the third pixel. Therefore, in the description of the present embodiment, the same portions as those in the first embodiment are denoted by the same reference numerals as those in FIGS.

FIG. 6 is a block diagram illustrating an example of a configuration of the NR processing unit 600 in the image processing unit 105.
As is clear from a comparison between FIG. 2 and FIG. 6, the NR processing unit 600 of the present embodiment has a configuration in which subtracters 601 and 602 and a comparator 603 are added to the NR processing unit 200 of the first embodiment. Having.

  In FIG. 6, the video signal Cur_in1 of the current frame and the video signal Mem_in1 of the previous frame before the current frame are output from the same one pixel (first pixel) on a predetermined horizontal line. The video signal Cur_in2 of the current frame and the video signal Mem_in2 of the previous frame before the current frame are output from the same pixel (second pixel) adjacent to the first pixel in one vertical direction. is there. These are the same as in the first embodiment.

  The video signal Cur_in3 of the current frame and the video signal Mem_in3 of the previous frame before that are the same pixel (third pixel) adjacent to the first pixel in the other vertical direction (opposite to the second pixel). Of the pixel).

  The video signals Cur_in2 and Cur_in3 of the current frame are video signals obtained with the same shutter speed (exposure time) and the same ISO sensitivity gain adjustment amount by the function of the stacked CMOS sensor. That is, Et1 and G1 shown in Expressions (1) and (2) are set as the exposure time and the gain for the first pixel, respectively. On the other hand, Et2 and G2 shown in the equations (1) and (2) are set as the exposure time and the gain for the second pixel and the third pixel, respectively.

Under the above settings, the NR processing section 600 performs the following processing.
First, with respect to the first pixel, the subtractor 201 subtracts the video signal Mem_in1 of the previous frame from the current frame video signal Cur_in1 to obtain a difference signal Δ1. Further, for the second pixel, the subtractor 203 subtracts the video signal Mem_in2 of the previous and previous frames from the video signal Cur_in2 of the current frame to obtain a difference signal Δ2.
Then, the difference signal Δ2 is subtracted from the difference signal Δ1 by the subtractor 204 to obtain a difference signal Δ12.
The above is the same as in the first embodiment.

  In the present embodiment, the subtracter 601 further subtracts the previous frame video signal Mem_in3 from the current frame video signal Cur_in3 from the video signal Cur_in3 of the third pixel to obtain a difference signal Δ3. In the present embodiment, the fifth difference signal deriving process for deriving the fifth difference signal in this manner is executed. Then, the difference signal Δ3 is subtracted from the difference signal Δ1 by the subtractor 602 to obtain a difference signal Δ13. In the present embodiment, the sixth difference signal deriving process for deriving the sixth difference signal in this way is executed.

  When the first pixel, the second pixel, and the third pixel receive the same amount of light reflected from the subject, the difference signals Δ12 and Δ13 (the absolute values thereof) are substantially the same (| Δ12 | ≒). | Δ13 |). in this case. Even if any of the difference signals Δ12 and Δ13 is multiplied by the cyclic coefficient K, appropriate NR processing can be performed.

  However, as shown in FIG. 7, when only the video signal Cur_in2 of the current frame of the second pixel has a different amount of light reflected from the subject, the difference signal Δ12 (the absolute value thereof) becomes the difference signal Δ13 ( Absolute value) (| Δ12 |> | Δ13 |). For example, such a case may occur when the edge 701 of the subject image is at the boundary between the first pixel and the second pixel. FIG. 7 shows five pixels on a vertical line (one rectangle indicates one pixel).

  At this time, in the first pixel, there is a case where there is no change between the video signal Cur_in1 of the current frame and the video signal Mem_in1 of the previous frame before that (that is, there is no edge of the subject image). In this case, when the cyclic coefficient K is multiplied by the difference signal Δ12 and the cyclic NR processing is performed as in the first embodiment, the NR processing becomes excessive. As a result, appropriate NR processing is not performed.

Therefore, in the present embodiment, the comparator 603 compares the output levels of the difference signals Δ12 and Δ13, selects the smaller one of the output levels of the difference signals Δ12 and Δ13, and outputs the smaller one to the multiplier 202. In the present embodiment, the first selection processing is executed in this way. Then, as described in the first embodiment, the multiplier 202 multiplies the difference signal Δ12 or Δ13 output from the comparator 603 by the cyclic coefficient K and outputs the result to the subtractor 205. The subtracter 205 subtracts the signal output by the multiplier 202 from the video signal Cur_in1 of the current frame of the first pixel, and sets the result as the output of the NR processing unit 600.
As described above, in addition to the effects described in the first embodiment, it is possible to prevent the deterioration of the image quality such as the deterioration of the resolution due to the execution of the inappropriate NR processing at the edge portion of the subject image or the like. NR processing can be performed more accurately.

(Third embodiment)
Next, a third embodiment will be described. In the first and second embodiments, for each of the first pixel, the second pixel, and the third pixel, difference signals Δ1 and Δ2 obtained by subtracting the video signal of the previous frame from the video signal of the current frame. , Δ3 were derived. Then, by subtracting the difference signals Δ2 and Δ3 for the second and third images from the difference signal Δ1 for the first pixel, noise and motion of the first image are distinguished.

  On the other hand, in the present embodiment, the motion of the subject image is detected from the signal level of the difference image obtained by subtracting the video signal of the current frame of the second pixel from the video signal of the current frame of the first pixel. Is changed (set) according to. A value obtained by multiplying the difference signal obtained by subtracting the video signal of the previous frame of the first pixel from the video signal of the current frame of the first pixel by the cyclic coefficient K as the value of the first pixel, Filter processing is performed using the value of the first pixel and pixels around the first pixel.

  As described above, the present embodiment and the first embodiment mainly differ in part of the NR processing. Therefore, in the description of the present embodiment, the same portions as those of the first and second embodiments are denoted by the same reference numerals as those shown in FIGS.

FIG. 8 is a block diagram illustrating an example of a configuration of the NR processing unit 800 in the image processing unit 105.
The NR processing section 800 includes a first NR processing section 810 and a second NR processing section 820. First NR processing section 810 performs first NR processing (cyclic NR processing) on the video signal input to NR processing section 800. The second NR processing unit 820 performs the second NR processing using the video signal on which the first NR processing has been performed.

  The subtractor 201 subtracts the video signal Mem_in1 of the previous and previous frames from the video signal Cur_in1 of the current frame to be processed to obtain a difference signal. The value obtained by multiplying the difference signal by the cyclic coefficient K by the multiplier 812 is set as a noise component, and the noise component is subtracted by the subtractor 205 from the video signal of the current frame Cur_in1 to realize noise removal. At this time, the cyclic coefficient K is derived using the result of the determination as to whether or not the subject image has moved by the motion detection processing.

As in the first and second embodiments, also in this embodiment, a stacked CMOS sensor is used for the image sensor 103. As described above, the stacked CMOS sensor has a function of changing the shutter speed (exposure time) and the ISO sensitivity gain adjustment amount for each horizontal line.
In FIG. 8, the video signal Cur_in1 of the current frame and the video signal Mem_in1 of the previous frame before the current frame are output from the same one pixel (first pixel) on a predetermined horizontal line. The video signal Cur_in2 of the current frame is an output from a pixel (second pixel) vertically adjacent to the first pixel.

  An example of the arrangement of the pixels on the vertical line is as described in the first embodiment. In the present embodiment, as in the first embodiment, a total of four pixels including one R pixel, one B pixel, and two G pixels shown in FIG. I do. Note that the arrangement of the pixels on the vertical line is not limited to that shown in FIG. 3A, as described in the first embodiment (for example, see FIG. 3B).

The first pixel and the second pixel in a unit of one pixel as described above are pixels on a horizontal line adjacent to each other in the vertical direction. With the function of the stacked CMOS sensor, the shutter speed (exposure time) and the ISO sensitivity gain adjustment amount can be changed between the first pixel and the second pixel.
Assuming that the exposure time and the gain for the first pixel are Et1 and G1, respectively, and the exposure time and the gain for the second pixel are Et2 and G2, respectively, in this embodiment, the following equations (4) and (5) It has the relationship of the formula.
(Time required to generate one frame of moving image) ≧ Et1> Et2 (5)
G1 <G2 (6)

The exposure times Et1 and Et2 and the gains G1 and G2 are read out from the ROM or RAM in the digital camera 100 and set by the AE process performed by the system control unit 102 at the time of capturing a moving image.
The above-described exposure times Et1 and Et2 and gains G1 and G2 are defined by the following equation (7) when the first pixel and the second pixel receive the same amount of light Y (= Y0). It is decided to be.
Et1 × G1 × Y0 = Et2 × G2 × Y0 (7)
The expression (7) is the same as the expression (3) shown in the first embodiment.

Therefore, when the subject is stationary, the amount of received light Y of the reflected light from the subject at the first pixel 901 and the second pixel 902 is the same (Y = Y ′). Therefore, from the following equations (8) and (9), there is a relationship of equation (10). That is, the areas of the regions 904 and 905 shown in FIG.
Cur_in1 = Et1 × G1 × Y ′ (8)
Cur_in2 = Et2 × G2 × Y ′ (9)
｜| Cur_in1−Cur_in2 | = 0 (10)
In FIG. 9, when the subject is stationary, the first pixel 901 and the second pixel 902 do not straddle the edge of the subject image and are pixels in the subject image.

On the other hand, when the subject is moving, since the exposure time and the gain are different, the amounts of light received from the subject at the first pixel 901 and the second pixel 902 are different, and the following equation (11) is used. Shows the relationship. That is, the area of the region 906 shown in FIG. 9 exceeds the area of the region 907, and these areas are different.
| Cur_in1−Cur_in2 |> 0 (11)
In FIG. 9, when the subject is moving, the first pixel 901 does not straddle the edge of the subject image and is a pixel in the subject image. On the other hand, the second pixel 902 is a pixel including the edge of the subject image.
In FIG. 9, x indicates the horizontal coordinate (position).

  From the above, in the present embodiment, the difference (= | Cur_in1−Cur_in2 |) between video signals from vertically adjacent pixels having different shutter speeds (exposure times) and ISO sensitivity gain adjustment amounts (gains). ) Is detected. In this way, it can be determined whether the subject is stationary or moving.

Actually, a noise component is superimposed on the video signal Cur_in1 of the first pixel of the current frame and the video signal Cur_in2 of the second pixel of the current frame. Therefore, in consideration of the uncertain element of the noise component, the presence or absence of the movement of the subject is determined by the following equations (12) and (13). In the expressions (12) and (13), ΔL is a noise component.
When | Cur_in1−Cur_in2 | ≦ ΔL → state in which the subject is stationary (12)
When | Cur_in1−Cur_in2 |> ΔL → The subject is moving (13)

  The noise component ΔL depends on the gain (setting of ISO sensitivity) of the first pixel and the second pixel and the brightness of the subject. Such a noise component ΔL is read out from the ROM or RAM in the digital camera 100 and set by the AE processing by the system control unit 102 at the time of capturing a moving image.

  The cyclic coefficient K is derived using the determination result of the presence or absence of the motion by the above-described motion detection processing, and is used for the first NR processing (cyclic NR processing). Specifically, when it is determined that the subject is in a stationary state, the NR effect is strengthened by setting the cyclic coefficient K to a relatively large value of 0 (zero) or more. On the other hand, when it is determined that the subject is moving, the NR effect is weakened by setting the cyclic coefficient K to a relatively small value of 0 (zero) or more. As a result, for a moving image area, the cyclic coefficient K of the cyclic NR processing is sufficiently small, so that occurrence of an afterimage can be suppressed. On the other hand, for a still image region, it is possible to sufficiently set the recursive coefficient K of the recursive NR process to obtain a sufficient NR effect, and to accurately execute the recursive NR process. Can be.

  In the present embodiment, in order to realize the above function, the subtractor 811 subtracts the video signal Cur_in2 of the second pixel of the current frame from the video signal Cur_in1 of the first pixel of the current frame. In the present embodiment, the seventh difference signal deriving process for deriving the seventh difference signal in this way is executed. The multiplier 812 determines the presence or absence of a motion of the subject in accordance with the magnitude relationship between the absolute value (| Cur_in1−Cur_in2 |) of the value and the predetermined value ΔL, and sets the cyclic coefficient K based on the determination result. I do. In the present embodiment, the detection processing for detecting the movement of the subject in the image and the setting processing are thus performed.

  A multiplier 812 multiplies a difference signal obtained by subtracting the video signal Mem_in1 of the first pixel of the previous frame from the previous frame by the subtractor 201 from the video signal Cur_in1 of the first pixel of the current frame. In the present embodiment, the eighth difference signal deriving process for deriving the eighth difference signal and the second signal deriving process are thus performed. Then, the subtractor 205 subtracts a signal obtained by multiplying the difference signal by the cyclic coefficient K from the video signal Cur_in1 of the first pixel of the current frame, and outputs the result as the output of the first NR processing unit 810. In the present embodiment, the ninth difference signal deriving process for deriving the ninth difference signal is thus realized.

  Next, an example of the second NR process in which the NR process is further performed by using the video signal after the first NR process is performed will be described. In the second NR process, for example, a filter process is performed using the pixel value of the target pixel to be processed and the pixel values of the peripheral pixels. Examples of the filter used here include a smoothing filter, a Gaussian filter, a median filter, and the like.

  In the present embodiment, also in the second NR processing, the image area in the stationary state and the image area in the moving state are determined using the determination result of the presence or absence of the movement of the subject by the above-described motion detection processing. The parameters of the NR process 2 are changed. The parameters of the second NR process can adjust the strength of the NR effect, and will be described later in detail, including the parameters of the first NR process.

If the NR circuit 821 determines that the subject is in a stationary state, the NR circuit 821 performs a second NR process by setting a parameter with a relatively weak NR effect. On the other hand, when it is determined that the subject is in a moving state, the NR circuit 821 performs a second NR process by setting a parameter with a relatively strong NR effect. Accordingly, noise is accurately detected even in an image region determined to be in a moving state during the first NR process, that is, an image region in which noise is not sufficiently removed in the first NR process. Can be removed. Note that whether or not the subject is in a stationary state is determined by, for example, Expressions (12) and (13).
As the second NR process, for example, a filtering process is performed on the target pixel using the pixel values of the peripheral pixels of the target pixel.

Next, the first NR process and the second NR process will be described, in particular, the setting of parameters for each NR process.
In the present embodiment, the above-described two types of NR processing (first NR processing and second NR processing) are executed. Therefore, as setting parameters for the NR process, there are roughly two types of parameters, a parameter for the first NR process and a parameter for the second NR process. These parameters are stored, for example, in the ROM or RAM in the digital camera 100.

These two types of parameters have different set values for each ISO sensitivity, and also differ depending on the result of the above-described determination of the presence / absence of motion (at least whether or not the subject is in a stationary state). FIG. 10 is a diagram illustrating an example of parameters of the NR process.
In FIG. 10, NR on the vertical axis indicates the degree of the effect of NR. The greater the value of this NR in the + direction (the direction of the arrow in the figure), the higher the effect of the NR. The horizontal axis indicates the ISO sensitivity of the first pixel. The greater the ISO sensitivity is in the + direction (the direction of the arrow in the figure), the greater the sensitivity is set as the ISO sensitivity.

Here, an example of a method for evaluating the above-described NR effect will be described.
A moving image for noise evaluation is photographed, and image data obtained by performing only the first NR processing and image data obtained by performing only the second NR processing using digital image data immediately before the NR processing. get. Based on these two types of image data, the noise in each NR process is evaluated. The magnitude relationship between the NR effects in the two types of NR processing is determined based on the magnitude of the noise evaluation value.

  In the present embodiment, for example, a chart of a plane having a predetermined luminance for noise measurement and uniform luminance at each position is prepared, and a moving image obtained by shooting this chart is set as a moving image for noise evaluation. This chart is stationary, and the acquired moving image shows the chart in a stationary state. In the present embodiment, a value obtained by measuring a change in luminance with respect to a predetermined luminance in this chart and performing numerical conversion is used as an evaluation value of noise. In this case, the smaller the numerical value of the luminance variation, the better the noise evaluation and the higher the NR effect.

  Further, the evaluation value of the noise due to the cyclic NR processing in the first NR processing tends to be saturated as the number of frames increases. Therefore, it is preferable to use image data of a frame in which the evaluation value of the noise by the first NR process is sufficiently saturated as the image data for which the noise is evaluated. In the present embodiment, the image data of the 60th frame is used as image data for noise evaluation.

  By the evaluation of the NR effect as described above, graphs 1001, 1002, 1003, and 1004 shown in FIG. 10 are obtained as the relationship between the parameters of the NR processing. A graph 1001 shows parameters with a relatively high NR effect in the first NR process, and a graph 1002 shows parameters with a relatively low NR effect in the first NR process. Further, a graph 1003 shows parameters of a setting where the effect of NR in the second NR processing is relatively high, and a graph 1004 shows parameters of a setting where the effect of NR in the second NR processing is relatively low.

  By performing the two NR processes indicating the relationship between the above parameters, a moving image having a time shorter than the exposure time Et1 (Et2 ≦ T ≦ Et1) despite being the exposure time Et1 of the first pixel as a moving image. The movement of the subject can be detected and reflected in the NR processing. That is, when the exposure time Et1 is the time required to generate one frame of the moving image, the movement can be detected even for the movement of the subject faster than the frame rate, and the NR process can be executed accurately. .

  Therefore, the motion of the subject can be detected at a higher speed than in the conventional case where the motion detection of the difference between frames is performed. Therefore, occurrence of an afterimage can be suppressed, and a moving image with good noise characteristics can be obtained.

(Fourth embodiment)
Next, a fourth embodiment will be described.
In the present embodiment, the exposure times Et1, Et2 and the gain of the first pixel and the second pixel in the first NR process are set as the following equations (14) and (15), respectively.
Et1 <Et2 <(time required to generate one frame of moving image) (14)
G1> G2 (15)
Except for this, the third embodiment is the same as the third embodiment.
By performing the NR processing with the above settings, the output of the first pixel and the second pixel used for detecting the presence or absence of motion can be obtained in a time Et2 shorter than the time required for generating one frame of the moving image.
In the present embodiment, the determination of the presence / absence of motion is made based on the time from the acquisition of the output of the first pixel and the second pixel used for detection of the presence / absence of motion to the timing of the next frame set at the frame rate. Then, it is possible to execute a cyclic NR process based on the determination of the presence or absence of a motion.
For this reason, in the NR processing for the current frame, it is possible to set the parameters of the NR processing based on the determination result of the presence or absence of the motion by the current frame. Therefore, NR processing can be realized quickly, with high accuracy, and with good performance.

(Fifth embodiment)
Next, a fifth embodiment will be described. In the third and fourth embodiments, the video signal of the second pixel adjacent to the first pixel in one vertical direction has been described as an example. On the other hand, in the present embodiment, one of the video signal of the second pixel adjacent to the first pixel in one direction in the vertical direction and the video signal of the third pixel adjacent to the first pixel in the other direction is used. Choose As described above, the present embodiment is different from the third and fourth embodiments mainly in the configuration and processing by using the video signal of the third pixel. Therefore, in the description of the present embodiment, the same portions as those in the first to fourth embodiments are denoted by the same reference numerals as those in FIGS.

FIG. 11 is a block diagram illustrating an example of a configuration of the NR processing unit 1110 in the image processing unit 105.
As is clear from a comparison between FIG. 8 and FIG. 11, the NR processing unit 1100 of the present embodiment is different from the NR processing units 800 of the third and fourth embodiments in that the first NR processing unit 1110 performs subtraction. It has a configuration in which a device 1111 and a comparator 1112 are added.
In FIG. 11, the video signal Cur_in1 of the current frame and the video signal Mem_in1 of the previous and previous frames are output from the same one pixel (first pixel) on a predetermined horizontal line. The video signal Cur_in2 of the current frame is an output from one pixel (second pixel) adjacent to the first pixel in one vertical direction. These are the same as the third and fourth embodiments.

The video signal Cur_in3 of the current frame is an output from one pixel (third pixel) adjacent to the first pixel in the other direction in the vertical direction (opposite to the second pixel).
The video signals Cur_in2 and Cur_in3 of the current frame are video signals obtained with the same shutter speed (exposure time) and the same ISO sensitivity gain adjustment amount by the function of the stacked CMOS sensor. That is, Et1 and G1 shown in Expression (5), Expression (6), Expression (14), and Expression (15) are set as the exposure time and the gain for the first pixel, respectively. On the other hand, as the exposure time and the gain for the second pixel and the third pixel, Et2 and G2 shown in the equations (5), (6) or (14) and (15) are set.

Under the above settings, the NR processing unit 1100 performs the following processing.
The video signal Cur_in2 of the second pixel of the current frame is subtracted from the video signal Cur_in1 of the first pixel of the current frame by the subtractor 811 to obtain a difference signal between the video signals of the first and second pixels of the current frame. Derive. Further, the subtractor 1111 subtracts the video signal Cur_in3 of the third pixel of the current frame from the video signal Cur_in1 of the first pixel of the current frame to obtain the difference between the video signals of the first and third pixels of the current frame. Derive the signal. In the present embodiment, the tenth difference signal deriving process for deriving the tenth difference signal is thus realized.

The comparator 1112 derives the absolute value | Δ12 | = | Cur_in1−Cur_in2 | of the difference signal between the video signals of the first and second pixels of the current frame. The comparator 1112 derives the absolute value | Δ13 | = | Cur_in1−Cur_in3 | of the difference signal between the video signals of the first and third pixels of the current frame.
If the amount of light received from the subject in the first to third pixels is the same, | Δ12 | ≒ | Δ13 |, and the motion detection result is the same regardless of which difference is used. NR processing can be executed even if the presence or absence of motion is determined using the detection result of.

  However, as shown in FIG. 7, when only the video signal Cur_in2 of the current frame of the second pixel has a different amount of light received from the subject, the difference signal Δ12 (the absolute value thereof) becomes the difference signal Δ13. (Absolute value) (| Δ12 |> | Δ13 |). For example, such a case may occur when the edge 701 of the subject image is at the boundary between the first pixel and the second pixel.

  Then, in the case of an image in which the difference between the reflected light from the subject in the first pixel and the second pixel is large and the difference signal Δ12> ΔL, even if the subject is stationary, the difference The determination result of the presence / absence of the movement based on the signal Δ12 is “a state where there is a movement”. As a result, appropriate NR processing is not performed.

  Therefore, in the present embodiment, the comparator 1112 compares the output levels of the difference outputs Δ12 and Δ13, selects the smaller one of the output levels of the difference signals Δ12 and Δ13, and outputs the smaller one to the multiplier 812 and the NR circuit 821. . In the present embodiment, the second selection processing is executed in this way. Then, as described in the third embodiment, the multiplier 812 uses the difference signal Δ12 or Δ13 output from the comparator 1112 to indicate whether the subject is stationary or moving. Is determined.

  When the difference signal Δ13 is selected, “Cur_in2” becomes “Cur_in3” in the expressions (12) and (13). The NR circuit 821 also determines whether the subject is stationary or moving using the difference signal Δ12 or Δ13 output from the comparator 1112, and determines whether the subject is stationary or moving. Set the parameters according to the result. In the present embodiment, the detection processing for detecting the movement of the subject in the image is performed in this manner.

  As described above, in addition to the effects described in the third and fourth embodiments, the occurrence of an afterimage or the like is performed by executing the NR processing based on the determination result of the inappropriate motion at the edge portion or the like of the subject image. Image quality can be prevented from deteriorating. Therefore, more accurate NR processing can be performed.

  It should be noted that each of the above-described embodiments is merely an example of a concrete example for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features.

(Other Examples)
The present invention is also realized by executing the following processing. That is, first, software (computer program) that realizes the functions of the above embodiments is supplied to a system or an apparatus via a network or various storage media. Then, a computer (or CPU, MPU, or the like) of the system or apparatus reads and executes the computer program.

  100: Digital camera, 105: Image processing unit, 200/600/800/1100: NR processing unit

Claims (16)

An image processing apparatus that processes an image captured by an imaging unit capable of setting an imaging condition for each first direction of an image,
The pixel signal of the first pixel and the pixel signal of the second pixel of the image of the current frame imaged by the imaging unit, and the pixel of the first pixel of the image of the frame earlier than the current frame using a signal having a processing means for performing processing to reduce noise of the first pixel,
The first pixel and the second pixel are pixels whose positions in a direction perpendicular to the first direction are different from each other,
The image processing device according to claim 1, wherein the pixel signal of the first pixel and the pixel signal of the second pixel are pixel signals obtained under different imaging conditions. Pixel signal of the first pixel of the past frame image, to claim 1, wherein the pixel signal noise is less than the pixel signal of the first pixel of the image of the current frame The image processing apparatus according to any one of the preceding claims. The processing unit is configured to calculate a difference between a pixel signal of a first pixel of an image of a current frame captured by the imaging unit and a pixel signal of the first pixel of an image of a frame earlier than the current frame. First differential signal deriving means for deriving a first differential signal that is a signal;
A second difference that derives a second difference signal that is a difference signal between a pixel signal of the second pixel of the image of the current frame and a pixel signal of the second pixel of the image of the past frame. Signal deriving means;
Third difference signal deriving means for deriving a third difference signal that is a difference signal between the first difference signal and the second difference signal;
The third, based on the difference signal and the engaging speed of the first signal derivation means for deriving a signal for correcting the image signal of the first pixel,
Fourth difference signal deriving means for deriving a fourth difference signal that is a difference signal between a pixel signal of a first pixel of the image of the current frame and the signal derived by the first signal deriving means When the image processing apparatus according to claim 1 or 2, further comprising a. The first difference signal deriving unit, when deriving the first difference signal for the pixel signal of the first pixel of the image of the first frame as the image of the current frame, The pixel signal of the first pixel by deriving the signal of the second pixel of the image of the current frame or the fourth difference signal instead of the pixel signal of the first pixel of the image 4. The image according to claim 3 , wherein a signal different from the signal of the first pixel of the image of the current frame, the noise of which has been reduced by another means than that of reducing the noise of the current frame, is used. Processing equipment. The first difference signal deriving unit reduces noise of a pixel signal of the first pixel by deriving the fourth difference signal as a pixel signal of the first pixel of the image of the past frame. 5. The image processing apparatus according to claim 3 , wherein a pixel signal of the first pixel of the image of the past frame, in which noise is reduced by another unit, is used. 6. The processing unit is a pixel signal of an image of the current frame, the pixel signal of a third pixel having a position in a direction perpendicular to the first direction different from the first pixel and the second pixel. And a fifth difference signal deriving unit that derives a fifth difference signal that is a difference signal from a pixel signal of the third pixel of the image of the past frame,
Sixth difference signal deriving means for deriving a sixth difference signal that is a difference signal between the first difference signal and the fifth difference signal;
There is further provided first selection means for selecting any one of the fourth difference signal and the sixth difference signal according to the levels of the fourth difference signal and the sixth difference signal. And
Said first signal deriving means, based on said said first selected by the selecting means and the fourth difference signal or said sixth differential signal of the coefficients of the image signal of the first pixel The image processing apparatus according to any one of claims 3 to 5, wherein a signal that takes a difference is derived.   The image processing apparatus according to claim 1, wherein the imaging condition includes an exposure time and a gain for the pixel signal. The processing means derives a seventh difference signal which is a difference signal between a pixel signal of a first pixel of the image of the current frame and a pixel signal of the second pixel of the image of the current frame. Seventh difference signal deriving means;
2. The image processing apparatus according to claim 1, further comprising: detecting means for detecting a motion of a subject in the image based on a level of the seventh difference signal. It said processing means includes setting means for setting the engagement speed based on the detection result by the detection means,
Deriving an eighth difference signal that is a difference signal between a pixel signal of a first pixel of the image of the current frame and the first pixel of an image of a frame earlier than the current frame. Difference signal deriving means,
On the basis of the engagement speed and set by the eighth differential signal and said setting means, and the second signal derivation means for deriving a signal having a difference between the image signal of the first pixel,
Ninth difference signal deriving means for deriving a ninth difference signal which is a difference signal between a pixel signal of a first pixel of the image of the current frame and the signal derived by the second signal deriving means The image processing apparatus according to claim 8, further comprising: A filter processing unit configured to filter the ninth difference signal derived by the ninth difference signal deriving unit by using the pixel values of peripheral pixels of the pixel of interest by the detection unit. Have more,
The filter processing means performs the filter processing such that noise is reduced more when the subject in the image is moving than when the subject in the image is stationary. The image processing device according to claim 9.   The apparatus according to claim 8, wherein the imaging condition includes an exposure time and a gain for a pixel signal. The product of the exposure time and the gain for the pixel signal of the first pixel is the same as the product of the exposure time and the gain for the pixel signal of the second pixel,
The detecting means determines that the subject is stationary if the absolute value of the level of the seventh difference signal is smaller than a predetermined value, and otherwise determines that the subject is moving. The image processing apparatus according to claim 11, wherein: The processing means includes a pixel signal of a first pixel of the image of the current frame and a pixel signal of an image of the current frame, wherein a position in a direction perpendicular to the first direction is the first signal. A tenth difference signal deriving unit that derives a tenth difference signal that is a difference signal between a pixel and a pixel signal of a third pixel different from the second pixel;
A second selection unit that selects any one of the seventh difference signal and the tenth difference signal according to the levels of the seventh difference signal and the tenth difference signal. And
The apparatus according to claim 1, wherein the detecting unit detects a movement of the subject in the image based on a level of the seventh differential signal or the tenth differential signal selected by the second selecting unit. The image processing device according to any one of claims 8 to 12. The product of the exposure time and the gain of the pixel signal of the first pixel is the product of the exposure time and the gain of the pixel signal of the second pixel, and the exposure time and the gain of the pixel signal of the third pixel. Is the same as any of
The detecting means determines that the subject is stationary if the absolute value of the level of the seventh differential signal or the tenth differential signal selected by the second selecting means is smaller than a predetermined value. 14. The image processing apparatus according to claim 13, wherein otherwise, it is determined that the subject is moving. An image processing method for processing an image captured by an imaging unit capable of setting an imaging condition for each first direction of an image,
The pixel signal of the first pixel and the pixel signal of the second pixel of the image of the current frame imaged by the imaging unit, and the pixel of the first pixel of the image of the frame earlier than the current frame using a signal, includes a processing step of performing processing for reducing the noise of the first pixel,
The first pixel and the second pixel are pixels whose positions in a direction perpendicular to the first direction are different from each other,
An image processing method, wherein the pixel signal of the first pixel and the pixel signal of the second pixel are pixel signals obtained under different imaging conditions.   A non-transitory computer-readable storage medium storing a program that causes a computer to function as each unit of the image processing apparatus according to claim 1.