JP6294596B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to a technique for suppressing noise of an image sensor in an imaging apparatus.

  In recent years, imaging devices such as home video cameras and digital still cameras have become popular. These imaging devices provide functions such as continuous shooting and moving image shooting, which improve convenience for users.

  However, image sensors used in these imaging devices contain a noise component specific to the element called dark current, and are greatly affected by dark current in high-sensitivity shooting, long-second shooting, and high-temperature shooting, and image quality deteriorates. Sometimes. As a countermeasure, there is a technique called blacking in which a light-shielded image is taken under the same conditions as in normal shooting and subtracted from the normal image to remove noise components. However, since black drawing needs to perform light-shielded shooting at the same exposure time as normal shooting for each shooting, there are problems that the shooting interval of continuous shooting becomes long and the frame rate of moving images decreases.

  On the other hand, Patent Document 1 proposes that a black image is taken immediately before (in advance) continuous shooting, and thereafter, blacking is performed using the black image (light-shielded image) taken in advance. .

JP 2012-130578 A

  However, the technique proposed in Patent Document 1 is based on the premise that the dark current component does not change during shooting from a black image shot in advance. However, in actual continuous shooting, if there is a temperature rise due to heat generation of the imaging device itself, there is a problem that dark current components increase and correct dark current noise removal cannot be performed.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to effectively reduce noise without increasing a shooting interval during continuous shooting such as continuous shooting or moving images in an imaging apparatus.

An imaging apparatus according to the present invention includes an imaging unit that captures a subject image formed by a photographic lens, an exposure control unit that controls an exposure condition of the imaging unit, and an image signal output from the imaging unit. A noise removing unit that removes noise from a signal output from a target pixel; a control unit that acquires a light-shielded image signal obtained in a state where the imaging unit is shielded before performing continuous shooting by the imaging unit; And the control means performs the noise removal on the noise removal means based on at least one of an exposure condition when performing continuous shooting, a temperature, and a degree of scratches present in the light-shielded image signal. The noise of the pixel signal to be performed is removed using the light-shielded image signal, or the noise is removed using a signal different from the light-shielded image signal. Controls whether to, said removal of the noise using a light-blocking image signals, a black subtraction process of subtracting the dark frame signal from the captured image signal by the image pickup means, separate from said light shielding image signal Noise removal using a signal is an interpolation process in which interpolation is performed using a signal of a peripheral pixel of a pixel to be subjected to the noise removal .

  According to the present invention, it is possible to effectively reduce noise without increasing the shooting interval during continuous shooting such as continuous shooting or moving images in the imaging apparatus.

1 is a diagram showing a configuration of a digital camera that is an embodiment of an imaging apparatus of the present invention. 6 is a flowchart illustrating a shooting operation of the digital camera according to the embodiment. The figure for demonstrating the flaw sort operation | movement in one Embodiment of this invention. The figure for demonstrating the flaw sort operation | movement in one Embodiment of this invention. The figure for demonstrating the flaw sort operation | movement in one Embodiment of this invention. The figure which showed the estimation method of a wound level. The figure which showed the process according to a wound level. The figure which showed the black drawing process. The figure which showed the flaw correction | amendment process. The figure which showed the real-time flaw detection and correction process.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a diagram illustrating a configuration of a digital camera which is an embodiment of an imaging apparatus of the present invention.

  In FIG. 1, reference numeral 100 denotes a digital camera which is an imaging apparatus. Reference numeral 10 denotes a photographing lens for forming a subject image, 12 denotes a mechanical shutter having a diaphragm function, 14 denotes an image sensor that converts an optical image into an electric signal and generates an image signal, and 16 denotes an analog signal output of the image sensor 14. It is an A / D converter for converting into a digital signal.

  A timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14 and the A / D converter 16, and is controlled by the memory control circuit 22 and the system control circuit 50. In addition to the mechanical shutter 12, the accumulation time can be controlled as an electronic shutter by controlling the reset timing of the image sensor 14 and can be used for moving image shooting and the like.

  An image processing circuit 20 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 16 or the data from the memory control circuit 22. Further, an electronic zoom function is realized by performing image cutting and scaling processing by the image processing circuit 20. The image processing circuit 20 performs predetermined calculation processing using the captured image data, and the system control circuit 50 controls the exposure control unit 40 and the distance measurement control unit 42 based on the obtained calculation result. TTL AF processing, AE processing, and EF processing are performed. Further, the image processing circuit 20 performs predetermined arithmetic processing using captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained arithmetic result.

  A memory control circuit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the memory 30, and the compression / decompression circuit 32. The data of the A / D converter 16 is written into the memory 30 via the image processing circuit 20 and the memory control circuit 22, or the data of the A / D converter 16 is directly written via the memory control circuit 22.

  Reference numeral 28 denotes an image display unit composed of a TFT LCD or the like. Display image data written in the memory 30 is displayed by the image display unit 28 via the memory control circuit 22. If the image data captured using the image display unit 28 is sequentially displayed, the electronic viewfinder function can be realized. The image display unit 28 can arbitrarily turn on / off the display according to an instruction from the system control circuit 50. When the display is turned off, the power consumption of the digital camera 100 can be significantly reduced. I can do it.

  Reference numeral 30 denotes a memory for storing captured still images and moving images, and has a storage capacity sufficient to store a predetermined number of still images and a moving image for a predetermined time. This makes it possible to write a large amount of images to the memory 30 at high speed even in continuous shooting or panoramic shooting in which a plurality of still images are continuously shot. The memory 30 can also be used as a work area for the system control circuit 50.

  Reference numeral 31 denotes a non-volatile memory composed of a flash ROM or the like. The program code executed by the system control circuit 50 is written in the nonvolatile memory 31, and the program code is executed while being read sequentially. In addition, an area for storing system information and an area for storing user setting information are provided in the non-volatile memory 31, and various information and settings are read and restored at the next startup.

  Reference numeral 32 denotes a compression / decompression circuit that compresses and decompresses image data by adaptive discrete cosine transform (ADCT) or the like. Write to. Reference numeral 40 denotes an exposure control unit that controls the mechanical shutter 12 having an aperture function, and has a flash light control function in conjunction with the flash 48. Reference numeral 42 denotes a distance measurement control unit that controls focusing of the photographing lens 10, and 44 denotes a zoom control unit that controls zooming of the photographing lens 10.

  A flash 48 has an AF auxiliary light projecting function and a flash light control function. The exposure control unit 40 and the distance measurement control unit 42 are controlled using the TTL method. Based on the calculation result obtained by calculating the captured image data by the image processing circuit 20, the system control circuit 50 performs the exposure control unit 40 and the distance measurement. The controller 42 is controlled.

  A system control circuit 50 controls the entire digital camera 100. Reference numerals 60, 62, 64, 66, 70, and 72 denote operation means for inputting various operation instructions of the system control circuit 50. A single unit such as a switch, a dial, a touch panel, pointing by line-of-sight detection, a voice recognition device, or the like Consists of multiple combinations.

  Here, a specific description of these operating means will be given. Reference numeral 60 denotes a mode dial switch, which can switch and set various function modes such as power-off, automatic shooting mode, shooting mode, panoramic shooting mode, moving image shooting mode, playback mode, and PC connection mode.

  Reference numeral 62 denotes a shutter switch SW1, which is turned on during the operation of the shutter button, and instructs to start operations such as AF (autofocus) processing, AE (automatic exposure control) processing, and AWB (auto white balance) processing. A shutter switch SW2 64 is turned on when the operation of the shutter button is completed. In the case of flash photography, after performing EF (flash pre-emission) processing, the image sensor 14 is exposed for the exposure time that is the exposure condition determined by the AE processing, and light is emitted during this exposure period, and the exposure period ends. At the same time, the exposure control unit 40 shields the light and ends the exposure to the image sensor 14. The shutter switch SW <b> 2 is a development process using a read process for writing a signal read from the image sensor 14 to the memory 30 via the A / D converter 16 and the memory control circuit 22, and an operation in the image processing circuit 20 or the memory control circuit 22. Processing, reading of image data from the memory 30, compression by the compression / decompression circuit 32, and an instruction to start a series of processing operations of recording processing to be written to the recording medium 200

  Reference numeral 66 denotes a display changeover switch, which can change the display of the image display unit 28. With this function, when photographing using the optical viewfinder 104, it is possible to cut power supply to the image display unit composed of a TFT LCD or the like to save power.

  Reference numeral 70 denotes an operation unit including various buttons, a touch panel, a rotary dial, and the like, which include a menu button, a set button, a macro button, a multi-screen playback page break button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, and the like. . Menu move + (plus) button, menu move-(minus) button, playback image move + (plus) button, playback image move-(minus) button, shooting quality selection button, exposure compensation button, date / time setting button, etc. There is also.

  A zoom switch 72 is used by the user to give an instruction to change the magnification of the captured image. The zoom switch 72 includes a tele switch that changes the imaging field angle to the telephoto side and a wide switch that changes the imaging angle of view to the wide angle side. The operation of the zoom switch 72 serves as a trigger for instructing the zoom control unit 44 to change the imaging field angle of the photographing lens 10 and performing an optical zoom operation. In addition, it also serves as a trigger for electronic zooming change of the imaging angle of view by image cropping by the image processing circuit 20 or pixel interpolation processing.

  Reference numeral 74 denotes a thermistor (temperature measurement sensor) for measuring the temperature inside the camera. Since defective pixels of the image sensor are affected by temperature, it is necessary to change the flaw correction processing depending on the temperature at the time of shooting. The thermistor is disposed near the image sensor in the image pickup apparatus, and measures the temperature of the image sensor itself.

  A power source 86 includes a primary battery such as an alkaline battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li ion battery, an AC adapter, and the like. Reference numeral 90 denotes an interface with a recording medium such as a memory card or hard disk, and reference numeral 92 denotes a connector for connecting to a recording medium such as a memory card or hard disk.

  Reference numeral 102 denotes a barrier that covers the imaging unit including the lens 10 of the digital camera 100 to prevent the imaging unit from being dirty or damaged. Reference numeral 104 denotes an optical viewfinder, which can take an image using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28.

A communication unit 110 has various communication functions such as USB, IEEE 1394, LAN, and wireless communication. Reference numeral 112 denotes a connector for connecting the image processing apparatus 100 to another device by the communication unit 110 or an antenna in the case of wireless communication.

  Reference numeral 200 denotes a recording medium such as a memory card or a hard disk. The recording medium 200 includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, or the like, an interface 204 with the digital camera 100, and a connector 206 for connecting with the digital camera 100.

  FIG. 2 is a flowchart illustrating a shooting operation of the digital camera according to the embodiment.

  When the shutter switch SW2 (64) is pressed, the system control circuit 50 starts the photographing operation. As the shutter speed, aperture value, and sensitivity for shooting, set values determined in advance by automatic exposure control or set values arbitrarily designated by the user are used.

  In S101, a black image (light-shielded image signal) is first photographed in a state where the image sensor 14 is shielded from light immediately before continuous photographing. The exposure and sensitivity settings for this image are the same as those for the main image to be taken (acquired). In S102, the flawed pixels present in the black image photographed in S101 are extracted and sorted by degree. This process will be described later in detail with reference to FIG. In order to assume a flawed pixel level of the image sensor 14, the output of the thermistor 74 is viewed and the exposure and sensitivity settings and temperature are stored.

  In S103, the main image is taken with the preset exposure / sensitivity settings. In S104, flaw correction is performed using the black image photographed in S101 and the flaw data sorted in S102. This process will be described later in detail with reference to FIG. In S105, it is determined whether or not the photographing instruction is continued. If the photographing is continued, the same processing is performed again from the main image photographing in S103.

  By not taking black images every time the main image is taken in this way, it is possible to shorten the shooting interval for main image shooting, to suppress the reduction in continuous shooting frame speed, and to reduce the reduction in the video frame rate. It becomes.

  3A to 3C are diagrams for explaining the flaw sorting operation in the present embodiment. As shown in the table of FIG. 3A, the scratches are graded according to degree. 12 bits: a table for taking a range of 0LSB to 4095LSB. Each pixel of the black image photographed in S101 is compared with the table in the table from Grade 0, and if the level of each Grade is exceeded, the Grade of that pixel becomes that Grade. The graph of FIG. 3B shows the Level change of Grade 0 to Grade 9. Grade 0 is a worse level of scratches, and Grade 9 is a Level that is less noticeable as a level of scratches.

  All pixels of the black image are graded, and pixels having better scratches than Grade 9 are determined not to be scratches, and are excluded from scratch correction targets. The damaged pixels of Grade 0 to 9 are sorted by grade, and the pixel addresses and levels are stored in the memory 30 in the format of FIG. 3C.

  FIG. 4 is a diagram showing a scratch level estimation method in the present embodiment. When the main image is captured in S103, the defective pixel level of the image sensor 14 is assumed. Therefore, the output of the thermistor 74 is viewed, and the signal of the flawed pixel whose signal level is black based on the exposure and sensitivity settings. Estimate how much the level has changed. FIG. 4 shows how much the signal level of the flawed pixel changes due to differences in temperature, exposure, and sensitivity when a black image is taken.

  The graph of “temperature vs. flaw level” shows changes in the signal level of flaw pixels due to temperature changes. A difference ΔTemp between the temperature at which the black image is taken and the temperature at which the main image is taken is obtained, and TempGain, which is a change in the signal level of the scratched pixel, is obtained from the graph.

  The graph of “ISOvs scratch level” shows a change in the signal level of the scratch pixel due to a change in ISO sensitivity. A difference ΔISO between the ISO sensitivity at which the black image is captured and the ISO sensitivity at which the main image is captured is obtained, and IsoGain, which is a signal level change degree of the damaged pixel, is obtained from the graph.

  The graph of “shutter speed vs. scratch level” shows the change in the signal level of the scratch pixel due to the change in the shutter speed. A difference ΔTv between the shutter speed at which the black image is captured and the shutter speed at which the main image is captured is obtained, and TvGain, which is a signal level change degree of the damaged pixel, is obtained from the graph.

By multiplying the signal level changes of these three kinds of scratched pixels, the final signal level change degree DefectGain of the scratched pixels is obtained by Expression (1).
DefectGain = TempGain × IsoGain × TvGain (1)
When DefectGain is obtained, the scratch data held when the black image is taken is converted by DefectGain, and the scratch correction method is changed for each grade. This process will be described later in detail with reference to FIG.

  FIG. 5 is a diagram showing processing according to scratch level in the present embodiment. Based on Grade and DefectGain, a scratch correction method is determined from the table. A indicates blacking, B indicates flaw correction, and C indicates real-time flaw correction. The scratch correction method is changed for each grade, and optimal scratch correction is performed.

  A scratch having a low level of scratch, that is, a grade having a low grade, is a very conspicuous scratch and has a large scratch level, so that it can be correctly corrected (noise removal) by black drawing. However, if DefectGain becomes high, the scratches themselves may be saturated and blacking may not be performed correctly, so switching to the scratch correction from the peripheral pixels of B is performed.

  Scratches with a moderate degree of damage, that is, a grade with medium grade are not immediately noticeable scratches and are not high in level, and may not be corrected correctly by black drawing. Scratch correction by interpolation from.

  Since a scratch having a good degree of scratch, that is, a grade having a high grade, is a scratch that is hardly noticeable, and there is a possibility that a pixel that is not a scratch is corrected by erroneous detection of noise, real-time scratch correction is used. That is, a flaw is detected in real time in the main image, and if it is a flaw, flaw correction is performed by interpolation from surrounding pixels.

  FIG. 6 is a diagram showing blacking processing in the present embodiment. The blacking process of the main image and the black image stored in the memory 32 is performed only for the pixels to be corrected. If the pixels other than the target are blacked, the S / N is deteriorated as a whole, and therefore unnecessary processing is not performed.

  Read out the pixel data to be blacked out for the main image and the pixel data to be blacked out for the black image and multiply the black image pixel by the previously obtained flaw level change degree DefectGain to convert the level of the flawed pixel. Then, subtraction is performed and setup addition is performed. This calculation is shown in Equation (2).

PixelDarkSub = PixelNormal-PixelDark x DefectGain + setup
... Formula (2)
This calculation is performed for all the target pixels, and the flaw correction by blacking is performed.

  FIG. 7 is a diagram showing a flaw correction process from peripheral pixels in the present embodiment. The center pixel in FIG. 7 will be described as a defect correction target. Since the target pixel is the pixel of the color filter “R”, scratch correction is performed using the peripheral “R” pixels. First, a level difference between diagonal pixels is obtained.

Diff1 = | R0 − R8 |
Diff2 = | R1 − R7 |
Diff3 = | R2 − R6 |
Diff4 = | R3 − R5 |
The minimum value is searched for among Diff1 to Diff4. If the difference is minimum, it can be said that there is no edge between the pixels, and correction by pixel interpolation is appropriate. An average value is obtained with a combination having the smallest difference, and is used as a signal for a flaw correction pixel.

  For example, when Diff1 is minimized, the signal Pix Correct for the flaw correction target pixel is obtained as follows.

PixCorrect = (R0 + R8) / 2
This is performed for all the target pixels, and flaw correction is performed by interpolation from surrounding pixels.

FIG. 8 is a diagram showing real-time flaw detection / correction processing in the present embodiment. In real-time flaw detection, it is determined whether the target pixel protrudes from the average level of surrounding pixels of the same color. If the “R” pixel in the center of FIG. 8 is a target pixel for real-time scratch detection, the maximum value PixMax and the minimum value PixMin of the surrounding R0 to R8 are obtained. If the difference MaxMinDiff between PixMax and PixMin is equal to or lower than the first predetermined level, the target pixel area does not have an edge portion, and real-time flaw determination is performed. If the difference PixDiff between the average value PixAve of R0 to R8 and the target PixTarget is equal to or higher than the second predetermined level, it is determined as a real-time scratch. That is,
MaxMinDiff = PixMax − PixMin
If MaxMinDiff is equal to or smaller than the first predetermined value, real-time scratch determination is possible. further,
PixDiff = PixTarget − PixAve
If PixDiff is equal to or greater than the second predetermined value, it is determined as a real-time scratch. If it is determined as a real-time flaw, flaw correction by interpolation from surrounding pixels is performed using the method described in FIG.

  In this embodiment, the three types of black correction, flaw correction by interpolation, and real-time flaw correction are used as the types of flaw correction. However, a simpler flaw correction method and a flaw correction that is detailed and takes a long processing time are used. May be. Moreover, although the grade is 10 stages, it may be further increased and combined with other flaw correction methods.

  In this embodiment, the degree of damage is estimated from the three parameters of temperature, sensitivity, and shutter speed, but other parameters such as an aperture, a zoom position, and flash emission may be used. The flaw correction method is not determined for each direction, but may be a method using an average value of all surrounding pixels.

Claims (8)

Imaging means for imaging a subject image formed by the taking lens;
Exposure control means for controlling exposure conditions of the imaging means;
Among the image signals output from the imaging unit, a noise removing unit that removes noise from a signal output from a target pixel;
Control means for acquiring a light-shielded image signal obtained in a state where the image pickup means is shielded before performing continuous shooting by the image pickup means;
The control means is configured to subject the noise removal means to the noise removal based on at least one of an exposure condition when performing continuous shooting, a temperature, and a degree of scratches present in the light-shielded image signal. Control whether to remove noise using the light-shielded image signal, or to remove noise using a signal different from the light-shielded image signal ,
The removal of the noise using the light-shielded image signal is a blacking process for subtracting the light-shielded image signal from the image signal picked up by the imaging unit, and noise removal using a signal different from the light-shielded image signal. Is an interpolation process for performing interpolation using a signal of a peripheral pixel of a pixel to be subjected to noise removal .   The imaging apparatus according to claim 1, wherein the exposure condition includes sensitivity of the imaging unit and shutter speed set by the exposure control unit.   The imaging apparatus according to claim 1, further comprising a sorting unit that extracts scratch pixels existing in the shading image signal and sorts the scratch pixels according to the degree of the scratch. For a scratched pixel having a level of scratches greater than a predetermined level, the signal level of the scratched pixel is converted, and correction is performed by blacking processing that subtracts the light-shielded image signal from the image signal captured by the imaging unit. The imaging apparatus according to claim 3 . With respect to a flawed pixel whose degree of flaw is greater than the predetermined level, if the converted signal level is higher than the second signal level as a result of the conversion of the signal level of the flawed pixel, the noise is removed. The imaging apparatus according to claim 4 , wherein correction is performed by interpolation using a signal of a peripheral pixel of the pixel. For scratched pixels having a degree of scratches larger than the first predetermined level and smaller than the second predetermined level larger than the first predetermined level, the signals of the peripheral pixels of the pixel to be subjected to the noise removal are used. The imaging apparatus according to claim 3 , wherein correction is performed by using interpolation. For a pixel having a degree of scratches greater than a predetermined level, it is determined from the image signal whether the pixel is a scratch pixel, and if it is determined to be a scratch pixel, interpolation is performed using signals from surrounding pixels. The imaging apparatus according to claim 3 , wherein correction is performed. A method for controlling an imaging apparatus comprising imaging means for imaging a subject image formed by a photographing lens,
An exposure control step for controlling an exposure condition of the imaging means;
Among the image signals output from the imaging means, a noise removal step for removing noise of signals output from the target pixels;
A control step of obtaining a light-shielded image signal obtained in a state where the image pickup means is shielded before performing continuous shooting by the image pickup means,
In the control step, an object to be subjected to the noise removal in the noise removal step based on at least one of an exposure condition when performing continuous shooting, a temperature, and a degree of scratches present in the light-shielded image signal. Control whether to remove noise using the light-shielded image signal, or to remove noise using a signal different from the light-shielded image signal ,
The removal of the noise using the light-shielded image signal is a blacking process for subtracting the light-shielded image signal from the image signal picked up by the imaging unit, and noise removal using a signal different from the light-shielded image signal. Is an interpolation process for performing interpolation using a signal of a peripheral pixel of a pixel to be subjected to noise removal, and a method for controlling the imaging apparatus.

Images