JP3950864B2 - Defective pixel correction apparatus and method - Google Patents

Description

  The present invention relates to a technique for correcting pixel defects in an image sensor.

  In an image sensor such as a CCD, it is known that local sensitivity failure of a semiconductor occurs in a manufacturing process or after manufacturing among pixels arranged in two dimensions. When these phenomena occur, it becomes impossible to obtain a charge output according to the amount of incident light, so that it appears as a so-called defective pixel in which white spots and black spots can be seen on the captured image regardless of the subject.

  Conventionally, in order to correct image quality degradation caused by imaging output of such defective pixels by signal processing, a defect signal for defective pixels included in an image sensor is detected in a semiconductor factory and stored in advance in a nonvolatile memory. At the time of shooting, defective pixel correction is performed based on the defective pixel signal of the nonvolatile memory. Alternatively, on the side of the image pickup apparatus using the image pickup device, when the shutter of the image pickup apparatus is in a light-shielded state, the pixel exceeding the predetermined output level (white point defective pixel) and the incident light amount to the image pickup apparatus are set to the predetermined level. In this case, the position signal of a pixel that does not reach the predetermined output level (black spot defective pixel) is stored in the nonvolatile memory, and the defective pixel is based on the position signal stored in the nonvolatile memory during normal imaging. And the defective pixel correction is performed by using, for example, the imaging output of the previous pixel instead of the imaging output of the defective pixel.

  By the way, the number of pixels required for an image sensor in recent years has increased from about several hundred thousand pixels to millions of pixels, and defective pixels appearing in the image sensor in spite of progress in manufacturing technology of the image sensor. The probability of occurrence tends to increase. In particular, in an image sensor used for a consumer device that requires low cost, in order to increase the manufacturing yield, the number of defective pixels has to be allowed to be much larger than the conventional one.

  Accordingly, it has become difficult to store a position signal of a defective pixel in an expensive non-volatile memory as in the prior art.

  In response to such a situation, for example, Japanese Patent Laid-Open No. 06-030425 (Patent Document 1) detects a level difference between pixel signals of a certain pixel and the same color pixel adjacent thereto during normal imaging, and performs predetermined processing. By determining the threshold value, the defective pixel is corrected, and the nonvolatile memory for storing the position information of the defective pixel in advance is omitted. Similarly, for example, in Japanese Patent Application Laid-Open No. 2002-027323 (Patent Document 2), a defective pixel included in a predetermined area of the image sensor is stored in a nonvolatile memory, and a defective pixel included outside the predetermined area is nonvolatile. After correcting defective pixels stored in the memory, detection and correction of defective pixels are performed simultaneously during normal imaging, so that even a limited non-volatile memory can handle a large number of defective pixels. .

  In addition, as the number of defective pixels is increased, the defective pixel correction has to be performed more. Therefore, there is a problem that the image quality deterioration due to the correction is naturally increased in the output image of the imaging apparatus.

For example, Japanese Patent Application Laid-Open No. 07-336602 (Patent Document 3) solves such a problem by comparing the sizes of boundaries between pixels around defective pixels and ranking the size of the boundaries. Even when a complex boundary exists in the vicinity, an optimum interpolation method is applied to each identified boundary pattern to prevent image quality deterioration due to correction.
Japanese Patent Laid-Open No. 06-030425 JP 2002-027323 A JP 07-336602 A

  However, in Japanese Patent Laid-Open No. 06-030425, a defective pixel correction is performed by detecting a level difference between pixel signals of a certain pixel and an adjacent same color pixel and performing a predetermined threshold determination during normal imaging. Therefore, naturally, high-frequency components such as subject edges are also erroneously detected and corrected. This hinders the image quality of the output image of normal pixels, and it cannot be said that this problem has been sufficiently studied. In Japanese Patent Application Laid-Open No. 2002-027323, defective pixels are determined and corrected in the normal imaging state except for a predetermined area, but the correspondence regarding erroneous detection of high-frequency components such as subject edges is completely taken into consideration. There wasn't.

  In Japanese Patent Laid-Open No. 07-336602, it is assumed that the position of the defective pixel has already been specified when identifying the boundary pattern around the defective pixel, and the position information of all the defective pixels that will continue to increase in the future is nonvolatile. The problem remains that it must be stored in advance in the memory. Further, even if a defective pixel is determined without using a non-volatile memory and a boundary pattern around the defective pixel is identified, if the peripheral pixel for identifying the boundary pattern includes a defective pixel, the boundary pattern Since the identification itself is not performed correctly, there is a problem that the effect aimed by this technique cannot be obtained at all.

  Accordingly, the present invention has been made in view of the above-described problems, and an object of the present invention is to enable correction of defective pixels effectively without causing deterioration in image quality.

In order to solve the above-described problems and achieve the object, a defective pixel correction apparatus according to the present invention detects defective pixel information of an image sensor including a plurality of pixels that capture an image of a subject and acquire an image signal. a defect pixel detection means and the image signal obtained from the image sensor, based on a detection result of the defect pixel detector, the edge direction detection means for detecting an edge direction of the object, a defective pixel of the imaging device Detecting information, correcting the output signal of the detected defective pixel by using the signals of the pixels in a plurality of directions, and generating a plurality of correction signals, and detection by the edge direction detecting means And selecting means for selecting one correction signal from the plurality of correction signals based on the result .

Further, the defective pixel correction method according to the present invention, a defect pixel detection step for detecting a defective pixel information of an image pickup device having a plurality of pixels to acquire an image signal by imaging an object image,
An image signal obtained from the image sensor, based on a detection result of the defect pixel detection step, the edge direction detection step of detecting an edge direction of the object, detecting a defective pixel information of the image sensor, is detected Based on the detection results of the defective pixel detection and correction step for generating a plurality of correction signals by performing correction of the output signals of the defective pixels using the signals of the pixels in a plurality of directions, respectively, And a selection step of selecting one correction signal from the correction signals .

  According to the present invention, it is possible to effectively correct a defective pixel without causing image quality degradation.

  Hereinafter, preferred embodiments of the present invention will be described.

(First embodiment)
FIG. 1 shows a first embodiment in which the present invention is applied to a black-and-white image pickup device or a defective pixel correction device that corrects white point defective pixels with respect to one color of a color image pickup device having an image pickup element for each color component such as an RGB3 plate. It is a figure which shows 1 embodiment.

  In FIG. 1, 1-1 is an image signal input terminal from an image sensor (not shown), 1-21 is a first line memory, 1-22 is a second line memory, 1-23 is a third line memory, -24 is a fourth line memory, 1-31 is a first defective pixel temporary detection unit, 1-32 is a second defective pixel temporary detection unit, 1-33 is a third defective pixel temporary detection unit, 1- 34 is a fourth defective pixel temporary detection unit, 1-35 is a fifth defective pixel temporary detection unit, 1-4 is a first threshold input terminal for temporary defective pixel detection, and 1-5 is in the edge direction of the image. An orthogonal luminance gradient direction detection unit, 1-6 is a two-dimensional defective pixel detection / correction unit, 1-7 is a second threshold input terminal for detecting a two-dimensional defective pixel, 1-8 is a changeover switch, and 1-9 is This is a corrected image output terminal.

  FIG. 2 is a detailed view of each defective pixel provisional detection unit 1-31 to 1-35 in FIG.

  2, reference numeral 2-1 denotes an image input terminal 1-1 connected to the image signal input terminal 1-1 or line memory outputs 1-21 to 1-24 shown in FIG. 1, and 2-2 denotes a first flip-flop circuit. Is a second flip-flop circuit, 2-4 is an addition circuit, 2-5 is a subtraction circuit, 2-6 is a limiter circuit, and 2-7 is a threshold input terminal connected to the first threshold input terminal 1-4 in FIG. , 2-8 are comparison circuits, 2-9 is a defective pixel provisional detection result output terminal, and 2-10 is an absolute value calculation circuit that can be used in place of the limiter 2-6.

  FIG. 3 is a detailed diagram of the luminance gradient direction detector 1-5 of FIG.

  In FIG. 3, 3-1 is a first image signal input terminal connected to the image signal input terminal 1-1 of FIG. 1, and 3-2 is a second image connected to the first line memory output 1-21 of FIG. A signal input terminal 3-3 is a third image signal input terminal connected to the second line memory output 1-22 in FIG. 1, and a third image signal input terminal 3-4 is connected to the third line memory output 1-23 in FIG. , 3-5 is a fifth image signal input terminal connected to the fourth line memory output 1-24 in FIG. 1, and 3-6 is a first defective pixel provisional detection unit 1-31 in FIG. The first temporary detection flag input terminal connected to the output of the second temporary detection flag input terminal 3-7, the second temporary detection flag input terminal connected to the output of the second defective pixel temporary detection unit 1-32 of FIG. A third temporary detection flag input terminal connected to the output of the third defective pixel temporary detection unit 1-33, 3-9 A fourth provisional detection flag input terminal 3-10 connected to the output of the fourth defective pixel provisional detection unit 1-34 in FIG. 1 and a third provisional output 3-10 connected to the output of the fifth defective pixel provisional detection unit 1-35 in FIG. 5 is a temporary detection flag input terminal.

  Reference numerals 3-11-1 to 3-11-5 denote flip-flop circuits that match the phases of the image signal input terminals 3-1 to 3-5 and the temporary detection flag input terminals 3-6 to 3-10, 3- 12-1 to 13-12-12 hold the peripheral pixel values for a total of 8 pixels, centering on the target pixel of luminance gradient direction detection, 2 pixels on the left and right in the horizontal direction, and 2 pixels on the top and bottom in the vertical direction. The flip-flop circuits 3-13-1 to 13-13-12 are centered on the target pixel of luminance gradient direction detection, 2 pixels on the left and right in the horizontal direction, and 2 pixels on the top and bottom in the vertical direction for a total of 8 pixels The flip-flop circuits 3-14-1 to 3-14-4 for holding the provisional detection flag have two upper and lower two pixels for each of the upper and lower two pixels in the vertical direction around the target pixel for luminance gradient direction detection. Add a logical expression according to the temporary detection flag value of the pixel AND elements 3-15-1 to 3-15-4 have the left and right provisional detection flag values for the left and right two pixels in the horizontal direction around the target pixel for luminance gradient direction detection. AND elements 3-16-1 to 3-16-3 for adding a corresponding logical expression are arithmetic circuits for calculating a differential value in the vertical direction around the target pixel for luminance gradient direction detection. 17-1 to 3-17-3 are arithmetic circuits for calculating a horizontal differential value centering on the target pixel for luminance gradient direction detection, and 3-18 is for switching the gain of the vertical differential value. A first switching unit, 3-19 is a second switching unit for switching the gain of the differential value in the horizontal direction, and 3-20 is a luminance gradient direction whose gain is adjusted by the switching units 3-18 and 3-19. Vertical fine around the target pixel Division circuit for computing the ratio of the values and the horizontal differential value, 3-21 table for calculating the inverse tangent function of the division circuit output 3-20 is 3-22 a brightness gradient direction signal output.

  FIG. 4 is a detailed view of the two-dimensional defective pixel detection / correction unit 1-6 of FIG.

  4, 4-1 is a first image signal input terminal connected to the output of the first line memory 1-21 of FIG. 1, and 4-2 is connected to the output of the second line memory 1-22 of FIG. The second image signal input terminal, 4-3 is a third image signal input terminal connected to the output of the third line memory 1-23 in FIG. 1, and 4-4-1 to 4-4-6 are defective pixels. This is a flip-flop circuit that holds the center pixel that is the object of detection and correction, and the image signals of eight neighboring pixels adjacent in the vertical, horizontal, and diagonal directions.

  4-6-1 is a first addition circuit that calculates an average value of upper and lower two pixels adjacent to each other in the vertical direction of the center pixel to be detected and corrected, and 4-6-2 is the center pixel and the center pixel. A first subtractor circuit for calculating a difference value from the average value of the upper and lower two adjacent pixels in the vertical direction, 4-6-3 is a first limiter circuit that sets 0 when the difference value is negative, 4-6 -4 is a first comparison circuit that compares the difference value processed by the first limiter circuit with the threshold value of the threshold value input 4-5, and outputs a High logic when the difference value is greater than the threshold value; 4-6-5 is a first switch for switching the center pixel value or the average value of the upper and lower two pixels adjacent in the vertical direction according to the output of the comparison circuit, and 4-6-6 is the first switch of the first switch. A first output terminal for outputting a corrected image of defective pixels in the vertical direction as an output It is.

  4-7-1 is a second addition circuit that calculates an average value of two left and right pixels adjacent in the horizontal direction of the center pixel to be detected and corrected, and 4-7-2 is the center pixel and the center pixel. A second subtractor circuit for calculating a difference value from the average value of the left and right two adjacent pixels in the horizontal direction; 4-7-3 is a second limiter circuit that sets 0 when the difference value is negative; 4-7 -4 compares the difference value processed by the second limiter circuit with the threshold value of the threshold value input 4-5, and outputs a High logic when the difference value is larger than the threshold value; 4-7-5 is a second switch for switching the central pixel value or the average value of the two left and right pixels adjacent in the horizontal direction in accordance with the output of the comparison circuit, and 4-7-6 is the second switch. A second output terminal for outputting a horizontal defective pixel corrected image as an output It is.

  4-8-1 is a third adder circuit that calculates an average value of two diagonal pixels adjacent to each other in the 45-degree direction of the center pixel to be detected and corrected, and 4-8-2 is the center pixel. And a second subtractor circuit 4-8-3 for calculating a difference value between the average value of two diagonal pixels adjacent in the 45-degree direction, and 4-8-3 is a third limiter circuit that sets 0 when the difference value is negative. 4-8-4 compares the difference value processed by the third limiter circuit with the threshold value of the threshold value input 4-5, and outputs a high logic when the difference value is larger than the threshold value. 4-8-5 is a third switch for switching the central pixel value or the average value of two diagonal pixels adjacent in the 45-degree oblique direction according to the comparison circuit output, and 4-8-6 Defect pixel corrected image in a 45 degree oblique direction, which is the output of the third switch Is a third output terminal.

  4-9-1 is a fourth addition circuit that calculates an average value of two diagonal pixels adjacent to each other in the direction of diagonal 135 degrees of the central pixel to be detected and corrected, and 4-9-2 is the central pixel. And 4-9-3 is a fourth limiter circuit that calculates 0 if the difference value is negative. 4-9-4 compares the difference value processed by the fourth limiter circuit with the threshold value of the threshold value input 4-5, and outputs a high logic when the difference value is larger than the threshold value. 4-9-5 is a fourth switch for switching the central pixel value or the average value of two diagonal pixels adjacent in the oblique 135 degree direction according to the output of the comparison circuit, and 4-9-6 Defect pixel correction in the oblique 135 degree direction which is the output of the fourth switch A fourth output terminal for outputting a completed image.

  FIG. 5 is a theoretical explanatory diagram of processing executed by the luminance gradient direction detection unit 1-5 in FIG.

  The operation of the first embodiment will be described with reference to FIGS. The image signal output from the image sensor including the defective pixel input from the image signal input terminal 1-1 in FIG. 1 is delayed by one horizontal period by each line memory 1-21 to 1-24, and the image signal is input. A total of five horizontal period image signals including the signal at the terminal 1-1 are generated. The image signals for the five horizontal periods are input to the luminance gradient direction detector 1-5 and the defective pixel temporary detectors 1-31 to 1-35.

  In the luminance gradient direction detection unit 1-5, in addition to the image signals for the five horizontal periods, each defective pixel temporary detection unit 1-31 to 1-35 has a first threshold input terminal 1 for temporary defective pixel detection. Temporary detection flags for five horizontal periods generated using the first threshold set at -4 are also input, and based on these, a luminance gradient direction signal is calculated for each pixel. On the other hand, image signals for a total of three horizontal periods (corresponding to three horizontal periods in the central portion of the image signals for the five horizontal periods), which are outputs of the line memories 1-21 to 1-23, are detected by two-dimensional defective pixel detection. -It inputs into the correction | amendment part 1-6.

  In the two-dimensional defective pixel detection / correction unit 1-6, pixel values for three pixels are generated for each of the image signals for the three horizontal periods, and a total of eight pixels that are adjacent to the center pixel and are vertically and horizontally oblique. And correcting the defect by detecting the defective pixel in the horizontal direction with respect to the center pixel using the second threshold value set from the second threshold value input terminal 1-7 for detecting the two-dimensional defective pixel. Generation of output, generation of correction output by detection of defective pixels in the vertical direction, generation of correction output by detection of defective pixels in the direction of 45 degrees diagonally, generation of correction output by detection of defective pixels in the direction of 135 degrees diagonally, ie, total A correction output is generated in four directions. The correction outputs in the four directions are sent to the switch 1-8, and one of the correction outputs is selected based on the luminance gradient direction signal, and is output from the corrected image output terminal 1-9.

  Each of the defective pixel provisional detection units 1-31 to 1-35 has exactly the same configuration, and the detailed operation of each will be described with reference to FIG.

  The image signal from the image input terminal 2-1 is delayed by the first flip-flop circuit 2-2 and the second flip-flop circuit 2-3, and a total of three pixels are held. The value of the first flip-flop circuit 2-2 is the center pixel, and the addition average of the image input signal 2-1 and the holding signal of the second flip-flop circuit 2-3, that is, the average of the left and right adjacent pixels of the center pixel Is calculated by the adder circuit 2-4, and a subtraction circuit 2-5 calculates a difference value between the central pixel and the average of the left and right adjacent pixels.

  In this embodiment, the purpose is to detect and correct white point defective pixels. Therefore, since the correction is not performed when the difference value is negative, limit processing is performed on the difference value in the limiter circuit 2-6. Although the limit processing is used in the present embodiment, the absolute value calculation circuit 2-10 can be used instead of the limiter circuit 2-6. In this case, not only the white point defect pixel but also the black point defect Pixel detection is possible. The difference value after the limit process is compared with the threshold value input 2-7. If the difference value after the limit process is larger than the threshold value, High logic is set to the defective pixel provisional detection result output 2-9, and the limit process is performed. When the subsequent difference value is less than or equal to the threshold value, a low logic is output.

  The detailed operation of the luminance gradient direction detection unit 1-5 and the processing contents thereof will be described with reference to FIGS.

  FIG. 5 is a diagram for explaining a method for obtaining the direction in which the luminance change of an image is most abruptly generated in a two-dimensional plane. As shown in FIG. 5, a reference axis (reference axis x) is defined for the two-dimensional image, and the direction in which the luminance changes most at any pixel of the two-dimensional image is the reference axis x. Can be expressed mathematically as follows using a spatial differentiation in two directions and a trigonometric function.

θ = arctan (diffY / diffX) (1)
diffY = dI (x, y) / dy (2)
diffX = dI (x, y) / dx (3)
Here, arctan is an inverse function of the tangent function tan, diffX is a spatial differential value of the image obtained along the reference axis x, and diffY is a second reference axis orthogonal to the reference axis x It is a spatial differential value of an image obtained along y. Each spatial differential value can be obtained by an FIR type digital filter represented by the following equation.

diff = a (−d0−d1 + d3 + d4) (4)
Here, a is a gain and is determined depending on the sampling frequency. However, in the present embodiment, a is not obtained in order to obtain the ratio of the differential values of two orthogonal axes. d 0, d 1, d 3, and d 4 are a total of four pixels adjacent to the left and right or the top and bottom, where d 2 not represented in Equation (4) is the central pixel. Further, although the calculation accuracy is lower than that of the equation (4), the spatial differential value can be obtained even with the following equation. It is necessary to double the gain due to the difference in the number of taps.

diff = 2a (-d1 + d3) (5)
diff = 2a (-d0 + d4) (6)
The angle θ obtained in this way can be calculated as a continuous value in the range of 90 ° to −90 ° with the reference axis x as 0 °. Therefore, for example, by calculating partial differential values in the horizontal direction and the vertical direction for the captured image, the luminance gradient direction θ with respect to the horizontal direction can be obtained. Since the edge direction of the image should be orthogonal to the luminance gradient direction (the slowest direction of luminance change), it can be determined uniformly from the angle θ.

  FIG. 3 is a configuration example of the calculation unit of the luminance gradient direction described with reference to FIG.

  Image signals of the image sensor including defective pixels for five horizontal periods input from the image signal input terminal 1-1 and the line memories 1-21 to 1-24 are input from the input terminals 3-1 to 3-5. The provisional detection flags for the five horizontal periods are input from the input terminals 3-6 to 3-10.

  When the temporary detection flag is generated in each defective pixel temporary detection unit in FIG. 2, a delay of one pixel occurs, so that each flip-flop circuit 3-11-1 for each of the image inputs 3-1 to 3-5. Are delayed by one pixel by ˜3-11-5, and the phases of the provisional detection flag and the image signal for the five horizontal periods are matched. After that, the image signals for the five horizontal periods are horizontally converted by the flip-flop circuits 3-12-1 to 13-12-12 in the horizontal direction with respect to the center pixel (3-12-6 flip-flop output). Each pixel (3-11-3, 3-12-5, 3-12-7, 3-12-8 flip-flop output), vertically two pixels (3-12-2, 3-12) -4, 3-12-10, and 3-12-12 flip-flop outputs), the peripheral pixel values for a total of 8 pixels are held. At the same time, the provisional detection flags corresponding to the peripheral pixels for the above eight pixels are also two pixels left and right in the horizontal direction with respect to the central pixel (3-13-6 flip-flop output) (3-8 provisional detection flag input, 3-13-5, 3-13-7, and 3-13-8 flip-flop outputs), two pixels in the vertical direction (3-13-2, 3-13-4, 3-13-10, 3-13-12 flip-flop outputs). Each of the addition / subtraction circuits 3-16-1 to 3-16-3 realizes the calculation of Expression (4) with respect to the adjacent pixels of two pixels above and below the central pixel, that is, a total of four pixels. In addition, the adder / subtracter circuits 3-17-1 to 3-17-3 perform the calculation of the expression (4) for the two adjacent pixels of the central pixel, that is, a total of four adjacent pixels.

  Here, in each of the adder circuits 3-16-1 and 3-16-2, a total of four pixels (3-12-2, 3-12-4, 3-12-10), each of the upper and lower two pixels with respect to the central pixel. , 3-12-12 flip-flop outputs) and provisional detection flags (3-13-2, 3-13-4, 3-13-10, 3-13-) respectively corresponding to the four pixels. A value obtained by ANDing the logical negation of each of the 12 flip-flop outputs) is input and added. The logical product is performed by AND elements 3-14-1 to 3-14-4. When the temporary detection flag is high logic, the corresponding pixel is determined to be a defective pixel, and therefore logical negation is performed before the logical product. As described above, by performing a logical product using the temporary detection flag, if any of d0, d1, d3, and d4 in equation (4) does not contain a defective pixel, the differential value in the vertical direction. diff is calculated by the equation (4) itself. If either d0 or d4 includes a defective pixel, d0 and d4 are forcibly set to 0 values, and as a result, the equation (5) ) To calculate the differential value diff. Similarly, when defective pixels are included in either d1 or d3, d1 and d3 are forcibly set to 0 values, and as a result, the differential value diff is calculated by equation (6).

  Here, neither d0 and d1, or d3 and d4 are assumed to be defective pixels.

  Similarly, in the horizontal direction, in each of the adder circuits 3-17-1 and 3-17-2, a total of four pixels (3-11-3, 3-12-5 , 3-12-7, 3-12-8 flip-flop outputs) and provisional detection flags (3-8 provisional detection flag inputs, 3-13-5, 3- 13-7 and 3-13-8 flip-flop outputs) and a logical product with the logical negation of the logical negation is input and added (the logical product is AND elements 3-15-1 to 3-15). -4). Thereby, any one of the equations (4) to (6) is adaptively selected depending on the presence / absence of the defective pixel, and the differential value diff is calculated.

  The calculation result of the differential value diff according to the equations (5) and (6) has a gain of 1/2 with respect to the calculation result according to the equation (4) due to the number of taps. Therefore, in each of the switching units 3-18 and 3-19, when the diff is calculated using the equation (5) or the equation (6), the switching is controlled so that the gain is doubled.

  The vertical differential value, which is the output of the switching unit 3-18, and the horizontal differential value, which is the output of the switching unit 3-19, are compared in the division circuit 3-20, and a table 3-21 is obtained. Is converted from the luminance gradient direction signal output terminal 3-22 as the value of the luminance gradient direction θ with respect to the horizontal direction. The luminance gradient direction signal can represent a total range of 180 degrees from 90 degrees to -90 degrees in the vertical direction, where the horizontal direction is 0 degrees.

  Detailed operation of the two-dimensional defective pixel detection / correction unit 1-6 in FIG. 1 will be described with reference to FIG.

  In FIG. 4, the image signals for the three horizontal periods (image signals input from the image signal input terminal 1-1 and the line memories 1- 1) input from the image signal input terminals 4-1 to 4-3. The image signals output from 21 to 1-24 correspond to the three horizontal periods of the central portion of the image signal for a total of five horizontal periods), and the flip-flop circuits 4-4-1 to 4-4-6 The pixel (4-4-3 flip-flop output) and the image signals for the eight neighboring pixels adjacent in the vertical, horizontal, and diagonal directions are held.

  First, two pixels (4-4-1 and 4-4-5 flip-flop outputs) adjacent to the upper and lower sides of the central pixel are added and averaged by the first adder circuit 4-6-1 to obtain a peripheral in the vertical direction. Calculated as the pixel average value. The average value of the peripheral pixels in the vertical direction is calculated by subtracting the average value of the peripheral pixels in the vertical direction from the central pixel by the first subtractor circuit 4-6-2, and the first limiter circuit 4-6-3. Will limit negative values to zero. This indicates that this embodiment is applied only to white point defect pixel correction.

  The first limiter output is compared with the threshold set at the threshold input terminal 4-5 in the first comparison circuit 4-6-4, and when the difference value is larger than the threshold, High logic is output. . Thus, in the first switch 4-6-5, when the output of the first comparison circuit is high logic, the first pixel is regarded as a defective pixel when the center pixel is viewed in the vertical direction. The average value of peripheral pixels in the vertical direction, which is the output of 6-1, is output as a correction value.

  When the first comparison circuit output is low logic, the center pixel is regarded as a normal pixel when viewed in the vertical direction, and the center pixel that is the output of the flip-flop circuit 4-4-3 is output as it is. In this way, a defective pixel correction output when the center pixel is viewed in the vertical direction is output from the first output terminal 4-6-6.

  Hereinafter, similarly, the defective pixel correction process in which the center pixel is seen in the horizontal direction is performed by the arithmetic circuits 4-7 to 4-7-5, limiter, comparison circuit, and switch, and the second output terminal 4. -7-6, and the defective pixel correction process in which the central pixel is seen in the oblique 45 degree direction is performed by each arithmetic circuit, limiter, comparison circuit, and switch of 4-8-1 to 4-8-5. 3 is output from the output terminal 4-8-6, and the defective pixel correction processing in which the central pixel is seen in the oblique 135 degree direction is performed by each arithmetic circuit, limiter, comparison circuit, and switch of 4-9-1 to 4-9-5. And output from the fourth output terminal 4-9-6.

  The switching control by the selector switch 1-8 in FIG. 1 will be described in detail.

  As described with reference to FIG. 4, the defective pixel correction output along the four directions of vertical, horizontal, diagonal 45 degrees, and diagonal 135 degrees with respect to the central pixel that is the defective pixel correction target is two-dimensional. Output from the defective pixel detection / correction unit 1-6 and input to the changeover switch 1-8. In addition, as described with reference to FIG. 3, the luminance gradient direction signal at the center pixel that is a defective pixel correction target is stably calculated regardless of the presence or absence of a defective pixel that may exist in the surrounding pixels. Used for control 1-8.

  Here, for example, when the luminance gradient direction signal indicates 0 degree, the luminance gradient of the central pixel changes most rapidly with respect to the horizontal direction, so the edge direction is the vertical direction, It is most desirable to select a defective pixel correction output along the vertical direction. Similarly, when the luminance gradient direction signal indicates 90 degrees or −90 degrees, it is desirable to select a defective pixel correction output along the horizontal direction, and when the luminance gradient direction signal is 45 degrees, When the defective pixel correction output is selected along the oblique 135 degree direction and the luminance gradient direction signal is −45 degrees, it is desirable to select the defective pixel correction output along the oblique 45 degree direction. As for the angle between these, naturally, an interpolation result along a direction closer to it may be selected. In this embodiment, since the edge direction is detected after the defective pixel is temporarily detected and temporarily corrected, an error in detecting the edge direction can be reduced. Further, since correction is performed in a correlated direction, highly accurate correction can be realized.

  With the above operation, in this embodiment, defective pixel correction along the edge direction of the subject can be operated stably and satisfactorily.

(Second Embodiment)
FIG. 6 shows a second example in which the present invention is applied to a color image pickup apparatus using a single-plate image pickup device of RGB (red, green, blue) Bayer array, particularly to a G pixel in which image quality deterioration is noticeable due to erroneous detection of defective pixels. It is a figure which shows embodiment.

  In FIG. 6, 6-1 is an image signal input terminal from an image sensor (not shown), 6-21 is a first line memory, 6-22 is a second line memory, 6-23 is a third line memory, 6 -24 is a fourth line memory, 6-31 is a first defective pixel temporary detection unit, 6-32 is a second defective pixel temporary detection unit, 6-34 is a third defective pixel temporary detection unit, 6- 35 is a fourth defective pixel temporary detection unit, 6-4 is a first threshold input terminal for temporary defective pixel detection, 6-5 is a defective pixel correction unit for R and B pixels, and 6-6 is an edge of an image. Luminance gradient direction detection unit orthogonal to the direction, 6-7 is a two-dimensional defective pixel detection / correction unit, 6-8 is a second threshold input terminal for two-dimensional defective pixel detection, 6-9 is a signal mixing unit, 6 -10 is a selector switch for G pixel and RB pixel, 6-11 is an identification signal input terminal for G pixel and RB pixel, 6-1 Is a corrected image output terminal.

  FIG. 7 is a detailed view of the luminance gradient direction detector 6-6 orthogonal to the edge direction of the image of FIG.

  In FIG. 7, 7-1 is a first image signal input terminal connected to the image signal input terminal 6-1 of FIG. 6, and 7-2 is a second image signal connected to the output of the first line memory 6-21 of FIG. An image signal input terminal, 7-4 is a third image signal input terminal connected to the output of the third line memory 6-23 in FIG. 6, and 7-5 is an output of the fourth line memory 6-24 in FIG. The fourth image signal input terminal connected, 7-6 is the first temporary detection flag input terminal connected to the output of the first defective pixel temporary detection unit 6-31 of FIG. 6, and 7-7 is the second of FIG. A second temporary detection flag input terminal connected to the output of the defective pixel temporary detection unit 6-32, and 7-9, a third temporary detection flag input connected to the output of the third defective pixel temporary detection unit 6-34 of FIG. A terminal 7-10 is a fourth temporary detection flag connected to the output of the fourth defective pixel temporary detection unit 6-35 in FIG. An input terminal.

  7-11-1 to 7-11-4 are flip-flop circuits that match the phases of the image signal inputs 7-1 to 7-5 and the temporary detection flag inputs 7-6 to 7-10. 1 to 7-12-7 are flip-flop circuits for holding G pixel values for a total of five pixels on two lines with respect to the target pixel of luminance gradient direction detection, and 7-12-8 to 7-12. −13 is a flip-flop circuit for holding G pixel values for a total of four pixels on one line with respect to the target pixel of the luminance gradient direction detection, and 7-12-14 to 7-12-19 are luminance gradients. Flip-flop circuit 7-12-20 to 7-12-26 for holding G pixel values for a total of four pixels of the line one line below the direction detection target pixel, 7-12-20 to 7-12-26 are luminance gradient direction detection target pixels G pixel value for a total of 5 pixels in the line 2 lines below Flip-flop circuits for holding, 7-13-1 to 7-13-5 are flip-flops for holding temporary detection flags corresponding to the G pixels of the line on two lines with respect to the target pixel of the luminance gradient direction detection. Circuits 7-13-8 to 7-13-11 are flip-flop circuits for holding temporary detection flags corresponding to the G pixels on one line with respect to the target pixel of the luminance gradient direction detection, 7-13. 14 to 7-13-17 are flip-flop circuits for holding temporary detection flags corresponding to the G pixels in the line one line below the target pixel for luminance gradient direction detection, and 7-13-20 to 7-13. Reference numeral 24 denotes a flip-flop circuit that holds a temporary detection flag corresponding to each G pixel in the line two lines below the target pixel for luminance gradient direction detection.

  7-14-1 to 7-14-4 perform correction processing in advance when any of the four G pixels adjacent in the oblique 135 ° direction is a defective pixel with respect to the target pixel of the luminance gradient direction detection. The correction block 7-14-5 to 7-14-8 is performed in advance when any of the four G pixels adjacent to the target pixel of the luminance gradient direction detection obliquely in the direction of 45 degrees is a defective pixel. It is a correction block for performing correction processing.

  Reference numerals 7-15-1 to 7-15-3 denote addition / subtraction circuits for performing a differential operation using a total of four G pixels adjacent in the oblique 135 degree direction, and 7-16-1 to 7-16-3 denotes the oblique operation. Addition / subtraction circuit for performing differential operation using a total of four G pixels adjacent in the 45 degree direction, 7-17 is a first absolute value calculation circuit, 7-18 is a second absolute value calculation circuit, and 7-19 is Comparison circuit, 7-20 is a first changeover switch, 7-21 is a second changeover switch, 7-22 is a division circuit, 7-23 is a luminance gradient direction flag output terminal, and 7-24 is a luminance gradient direction signal output. Terminal.

  FIG. 8 is a detailed view of each of the correction blocks 7-14-1 to 7-14-8 in FIG.

  In FIG. 8, 8-1 and 8-2 are peripheral pixel input terminals, 8-3 is a central pixel input terminal, 8-4 is a temporary detection flag input terminal, 8-5 is an adder circuit for calculating an addition average, 8- 6 is a changeover switch, and 8-7 is a correction output.

  FIG. 9 is a detailed view of the two-dimensional defective pixel detection / correction unit 6-7 in FIG.

  9, 9-1 is a first image signal input terminal connected to the image signal input terminal 6-1 in FIG. 6, and 9-2 is a second image connected to the output of the first line memory 6-21 in FIG. The signal input terminal, 9-3 is connected to the third image signal input terminal connected to the output of the second line memory 6-22 of FIG. 6, and 9-4 is connected to the output of the third line memory 6-23 of FIG. A fourth image signal input terminal 9-5 is a fifth image signal input terminal connected to the output of the fourth line memory 6-24 in FIG.

  9-6-1 to 9-6-14 are defective pixel detection and correction target pixels, four pixels adjacent in the vertical direction around the target pixel, four pixels adjacent in the horizontal direction, and This is a flip-flop circuit for holding a total of 13 pixels, two pixels adjacent in a 45-degree direction and two pixels adjacent in a 135-degree direction.

  9-7-1 is a first detection / correction circuit that detects and corrects a defective pixel in a direction of 135 degrees obliquely with respect to the target pixel, and 9-7-2 detects a defective pixel in a direction perpendicular to the target pixel. 9-7-3 is a third detection / correction circuit for detecting and correcting defective pixels in the horizontal direction with respect to the target pixel, and 9-7-4 is the target. This is a fourth detection / correction circuit for detecting and correcting defective pixels in a 45-degree oblique direction with respect to the pixels.

  9-8 is a threshold input terminal connected to the second threshold input terminal 6-8 for detecting a two-dimensional defective pixel in FIG. 6, 9-9 is a corrected image signal output terminal in a diagonal 135 degree direction, and 9-10 is a vertical direction. The corrected image signal output terminal 9-11 is a corrected image signal output terminal in the horizontal direction, and 9-12 is a corrected image signal output terminal in the oblique 45 degree direction.

  FIG. 10 is a detailed diagram of each of the detection / correction circuits 9-7-1 to 9-7-4 in FIG.

  In FIG. 10, 10-1 and 10-2 are image input terminals to which peripheral pixels adjacent to the defective pixel detection and correction target pixels are input, and 10-3 is an image to which the defective pixel detection and correction target pixels are input. An input terminal, 10-4 is a threshold input terminal, 10-5 is an addition circuit for calculating an addition average of the peripheral pixels, and 10-6 is a subtraction circuit for calculating a difference between the addition average and the target pixel value. 7 is an absolute value calculation circuit, 10-8 is a comparison circuit, 10-9 is a changeover switch, and 10-10 is a corrected image output terminal.

  FIG. 11 illustrates the mix control of each of the corrected image signals 9-9 to 9-12 in FIG. 9 according to the luminance gradient direction signal and the luminance gradient direction flag in the signal mixing unit 6-9 in FIG. FIG.

  The operation of the second embodiment of the present invention will be described with reference to FIGS.

  The output image signal of the image sensor including the defective pixel input from the image signal input terminal 6-1 in FIG. 6 is delayed by one horizontal period by each line memory 6-21 to 6-24, and the image signal is input. A total of five horizontal period image signals including the signal at the terminal 6-1 are generated. The image signals for the five horizontal periods are input to the two-dimensional defective pixel detection / correction unit 6-7, and the vertical, horizontal, diagonal 45 degrees, and diagonal 135 degrees are centered on the defective pixel detection and correction target pixels. Detection and correction of defective pixels along a total of four directions are performed, and respective corrected image outputs are output. The detailed operation will be described later with reference to FIGS.

  Next, among the image signals for the five horizontal periods, the output image signal of the second line memory 6-22, which is the center line thereof, is sent to the defective pixel correction unit 6-5 for R and B pixels. Detecting and correcting defective pixels is performed by the same processing contents as in the prior art. Among the image signals for the five horizontal periods, the image signals for a total of four horizontal periods other than the central line are orthogonal to the defective pixel temporary detection units 6-31 to 6-35 and the edge direction of the image. As will be described later with reference to FIGS. 7 and 8, a luminance gradient direction signal and a luminance gradient direction flag are generated in the target pixel for detection and correction of the defective pixel. Is done.

  The corrected image output along a total of four directions of the vertical, horizontal, diagonal 45 degrees, and diagonal 135 degrees, the luminance gradient direction signal, and the luminance gradient direction flag are sent to the signal mixing unit 6-9, which will be described later with reference to FIG. As described above, in accordance with the luminance gradient direction signal and the luminance gradient direction flag, the correction image output is mixed along the total of the four directions of the vertical, horizontal, oblique 45 degrees, and oblique 135 degrees to perform a two-dimensional defect. Output as pixel correction output. Since the two-dimensional defective pixel correction output is effective only for the G pixel in this embodiment, the two-dimensional defective pixel correction output is performed at the changeover switch 6-10 according to the identification signal input 6-11 for the G pixel and the RB pixel. The defective pixel correction unit output 6-5 for the R and B pixels is switched and output from the corrected image output terminal 6-12.

  Next, each element will be described. As shown in FIG. 9, the two-dimensional defective pixel detection / correction unit 6-7 in FIG. 6 receives the image signals for the five horizontal periods from the image signal input terminals 9-1 to 9-5. The flip-flop circuits 9-6-1 to 9-6-14 hold a plurality of pixels for each line. That is, the image signal delayed by two horizontal periods given from the third image signal input terminal 9-2 is a central line including the target pixel for defective pixel detection and correction. Here, the target pixel and its right and left 2 A total of 5 pixels are retained (9-3 input and 9-6-6 to 9-6-9 flip-flop output). The target pixel becomes the flip-flop output 9-6-7.

  With respect to the upper and lower lines of the central line including the target pixel, a total of three pixels are held (9-6-3 to 9-6) corresponding to the pixel at the same horizontal position as the target pixel and its left and right pixels. -5 flip-flop output and 9-6-10-9-6-12 flip-flop output). Further, for the upper and lower lines of the upper and lower lines, the pixels at the same horizontal position as the target pixel are held (the flip-flop outputs of 9-6-2 and 9-6-14). Reference numerals 9-6-1 and 9-6-13 are flip-flops for delay adjustment.

  As described above, the present embodiment is an embodiment in which the present invention is applied only to a color image pickup apparatus using an RGB Bayer array single-plate image pickup device, in particular, only to G pixels in which image quality deterioration is conspicuous due to erroneous detection of defective pixels. For this reason, when the target pixel is regarded as a G pixel, it is held by the image signal inputs 9-1 to 9-5 and the flip-flops 9-6-1 to 9-6-14. The G pixel is as follows.

  The G pixels adjacent in the vertical direction of the target pixel are the flip-flop outputs of 9-6-2 and 9-6-14, and the G pixels adjacent in the horizontal direction are the image signal input of 9-3 and 9 G-pixels that are -6-9 flip-flop outputs and are adjacent to each other at 45 degrees diagonally are the outputs of 9-6-10 and 9-6-5 flip-flops. These are the flip-flop outputs of 9-6-3 and 9-6-12.

  Of the pixel values held in the respective flip-flops in this way, the G pixel adjacent to the target pixel in the oblique direction of 135 degrees is sent to the first detection / correction circuit 9-7-1 in the direction perpendicular to the target pixel. G pixel adjacent to the second detection / correction circuit 9-7-2, and the G pixel adjacent to the target pixel in the horizontal direction to the third detection / correction circuit 9-7-3 G pixels that are adjacent in the 45-degree direction are sent to the fourth detection / correction circuit 9-7-4, and as shown in FIG. 10, the difference between the average value of the adjacent pixels and the target pixel in each of the four directions. Whether or not the pixel is defective is determined by the threshold processing of the value. If the pixel is determined to be defective, the adjacent pixel average value is output as a correction value instead of the target pixel. As described above, in each output of 9-9 to 9-12, an image signal in which the target pixel is detected and corrected in four directions of vertical, horizontal, 45 degrees oblique, and 135 degrees oblique is output.

  The detailed operation of the luminance gradient direction detection unit 6-6 in FIG. 6 will be described with reference to FIGS.

  In the luminance gradient direction detection unit, a total of four lines of image signals are provided from each of the image signal input terminals 7-1 to 7-5 except for the center line including the defective pixel detection and correction target pixels. Entered. The image signals for the four lines are first subjected to phase matching with a temporary detection flag (to be described later) by flip-flop circuits 7-11-1 to 7-11-4, and then each flip-flop circuit 7-12-1. Image signals for a plurality of pixels corresponding to each line are held at ˜7-12-26. In the luminance gradient direction detection unit, the target pixel for the defective pixel detection and correction is not input, but when the target pixel is a G pixel, each image signal input of 7-1 to 7-5 and each flip-flop Of the image signals held by the circuits 7-12-1 to 7-12-26, the same G pixel is as follows for each line.

  In a line two lines above the target pixel line, an image signal input of 7-1 and outputs of flip-flops of 7-12-1, 7-12-3, 7-12-5, 7-12-7, 7-11-2, 7-12-9, 7-12-11, 7-12-13 flip-flop output in the line one line above the target line, and in the line one line below the imaging line 7-11-3, 7-12-15, 7-12-17, 7-12-19 flip-flop outputs, and in the line 2 lines below the imaging line, 7-5 image signal input, 7-12-20, 7-12-22, 7-12-24, and 7-12-26 flip-flop outputs.

  In order to detect the luminance gradient direction in the target pixel, as described with reference to FIG. 5, it is necessary to calculate differential values along two reference axes. In this embodiment, the RGB Bayer array G It matches the characteristics of the pixel array.

  That is, since the correlation distance with the target pixel is shorter and the G pixel can be obtained on each line when viewing the target pixel in the oblique 45 degree direction and the oblique 135 degree direction, these directions are used as reference axes. It is supposed to be taken. Therefore, the G pixels used for calculating the differential value are four flip-flop outputs 7-12-5, 7-12-11, 7-12-15, and 7-12-20 in the 45-degree direction. In the diagonal direction of 135 degrees, four flip-flop outputs 7-12-1, 7-12-9, 7-12-17, and 7-12-24 are obtained. That is, the differential value is calculated using the four adjacent pixels in each direction with the target pixel as the center, and according to the equation (4), the above 7-12-5, 7-12-11, 7-12. 15, 7-12-20, and 7-12-1, 7-12-9, 7-12-17, 7-12-24. However, 7-12-5, 7-12-11, 7-12-15, 7-12-20, and 7-12-1, 7-12-9, 7-12-17, 7-12. Since a differential value cannot be calculated correctly if a defective pixel is included in each pixel that is each flip-flop output of −24, 7-12-5, 7-12-11, 7-12-15, 7 -12-20, and 7-12-1, 7-12-9, 7-12-17, 7-12-24 flip-flop outputs are not directly used for differential value calculation, The differential value calculation is performed after the correction processing using the two right and left G pixels adjacent in the horizontal direction is performed in each of the correction blocks 7-14-1 to 7-14-8.

  The determination as to whether or not the correction processing is performed in each correction block is given from each detection flag input 7-6 to 7-10, and each pixel held by 7-13-1 to 7-13-24 is determined. This is performed with reference to the corresponding temporary detection flag. The temporary detection flag is generated as described in FIG. 2 in the first embodiment, and becomes High logic when it is determined that the pixel is a defective pixel. However, in the present embodiment, by using the absolute value calculation circuit 2-10 in FIG. 2, the temporary detection flag is set to be high regardless of whether the pixel is a white point defective pixel or a black point defective pixel.

  The inputs in each correction block are summarized as follows. In the correction block 7-14-1, the output of 7-12-1, the image signal input of 7-1 which is a G pixel horizontally adjacent to the 7-12-1, and the output of 7-12-3. The output and the output of 7-13-1, which is a temporary detection flag corresponding to the above 7-12-1, are input. In the correction block 7-14-2, the output of 7-12-9, the output of 7-11-2 which is a G pixel adjacent to the 7-12-9 in the horizontal direction, and the output of 7-12-11 The output and the output of 7-13-9 which is a provisional detection flag corresponding to 7-12-9 are input. In the correction block 7-14-3, the output of 7-12-17, the output of 7-12-15 which is a G pixel adjacent to the 7-12-17 in the horizontal direction, and 7-12-19 The output and the output of 7-13-17, which is a temporary detection flag corresponding to 7-12-17, are input. In the correction block 7-14-4, the output of 7-12-24, the output of 7-12-22 which is a G pixel adjacent to the 7-12-24 in the horizontal direction, and 7-12-26 The output and the output of 7-13-24, which is a temporary detection flag corresponding to 7-12-24, are input. In the correction block 7-14-5, the output of 7-12-5, the output of 7-12-3 which is a G pixel adjacent to the 7-12-5 in the horizontal direction, and the output of 7-12-7 The output and the output of 7-13-5, which is a temporary detection flag corresponding to 7-12-5, are input. In the correction block 7-14-6, the output of 7-12-11, the output of 7-12-9 which is a G pixel adjacent to the 7-12-11 in the horizontal direction, and 7-12-13 The output and the output of 7-13-11, which is a temporary detection flag corresponding to the 7-12-11, are input. In the correction block 7-14-7, the output of 7-12-15, the output of 7-11-3 which is a G pixel adjacent to the 7-12-15 in the horizontal direction, and 7-12-17 The output and the output of 7-13-15, which is a temporary detection flag corresponding to the 7-12-15, are input. In the correction block 7-14-8, the output of 7-12-20, the input of an image signal of 7-5 which is a G pixel adjacent to the 7-12-20 in the horizontal direction, and the output of 7-12-22 The output and the output of 7-13-20, which is a temporary detection flag corresponding to 7-12-20, are input.

  The contents of the correction blocks 7-14-1 to 7-14-8 described above are all common and are as shown in FIG. That is, the G pixel adjacent in the horizontal direction is input from the peripheral pixel input terminals 8-1 and 8-2, and the G pixel to be subjected to the differential value calculation is input from the center pixel input terminal 8-3. The temporary detection flag is input from the temporary detection flag input terminal 8-4, 8-5 is an addition circuit for calculating an addition average, 8-6 is a changeover switch, and when the temporary detection flag is high, that is, the differentiation When the G pixel whose value is to be calculated is determined as a defective pixel, the addition average value from the adjacent G pixel is selected and output.

  As described above, the influence of defective pixels of the G pixel used for calculating each differential value is suppressed, and each of the addition / subtraction circuits 7-15-1 to 7-15-3 and 7-16-1 to 7-16-3. Accordingly, the calculation of the differential value calculation of the equation (4) is executed, and the two differential values in the oblique 45 degree direction and the oblique 135 degree direction are calculated. The absolute values of the two differential values are obtained in the absolute value calculation circuits 7-17 and 7-18, and in which direction the differential value is larger is determined in the comparison circuit 7-19.

  The determination result of the comparison circuit is output from the luminance gradient direction flag output terminal 7-23 and is used to control the changeover switches 7-20 and 7-21. In each of the change-over switches 7-20 and 7-21, the division circuit 7-22 at the subsequent stage performs switching control so that the denominator is in a state equal to or higher than the numerator. That is, when the magnitude of the differential value in the oblique 45 degree direction is larger than the magnitude of the differential value in the oblique 135 degree direction, the division circuit 7-22 switches so as to calculate the oblique 135 degree / diagonal 45 degree. In the opposite case, switching is performed so that 45 degrees / 135 degrees is calculated. As described above, when the calculation in the division circuit is made to always have a large denominator, even if the arctangent function of the equation (1) is omitted, the error of the calculation result is small, and the result of the division is directly used as the luminance gradient. It can be used as a direction signal. Further, since the denominator is always greater than or equal to the numerator, the luminance gradient direction signal has a value that falls within a range of ± 1. The luminance gradient direction signal thus obtained is output from the luminance gradient direction signal output terminal 7-24.

  Next, detailed operation of the signal mix unit 6-9 in FIG. 6 will be described.

  As the luminance gradient direction flag, when the magnitude of the differential value in the oblique 45 ° direction> the magnitude of the differential value in the oblique 135 ° direction, the flag = High logic, the signal mixing unit 6-6 in FIG. The operation in 9 is as follows.

  FIG. 11 conceptually illustrates the operation. First, the relationship between the luminance gradient direction flag and the luminance gradient direction input to the signal mixing unit is as follows. Means that the luminance gradient direction is 0 degree, and the flag indicates that Low logic = an oblique 135 degree direction (denoted as d2 in FIG. 11) is 0 degree luminance gradient direction. . Thus, for example, if the luminance gradient direction is exactly 45 degrees, the luminance gradient direction signal is 0 and the luminance gradient direction flag is high logic. Similarly, when the angle is 135 degrees, the luminance gradient direction signal is 0, and the luminance gradient direction flag is low logic.

  Next, between the oblique 45 degree direction and the vertical direction (denoted as V in FIG. 11), the luminance gradient direction flag has a high logic, and the luminance gradient direction signal has a value between 0 and 1. Similarly, the flag has a high logic between 45 degrees and a horizontal direction (indicated as H in FIG. 11), and the luminance gradient direction signal has a value between 0 and -1.

  Similarly, the flag is low and the luminance gradient direction signal is a value between 0 and 1 between the diagonal direction of 135 degrees and the vertical direction, and the flag is low and the gradient direction is between 135 degrees and the horizontal direction. The signal is a value between 0 and -1.

  Now, the signal mix unit 6-9 in FIG. 6 receives vertical, horizontal, and diagonal 45 pixels from the two-dimensional defective pixel detection / correction unit 6-7 in FIG. The corrected image signal corrected for defective pixels in four directions of 1 degree and 135 degrees is input, and the four corrected image signals are mixed to generate a two-dimensional corrected image signal. In this embodiment, Follow the instructions.

  That is, when the flag is high and the luminance gradient direction signal is a positive value, the luminance change in the target pixel for detection and correction of the defective pixel is most severe in any direction between 45 degrees and the vertical direction. Therefore, as the two-dimensional corrected image signal, the corrected image signal in the horizontal direction and the corrected image signal in the oblique 135 degree direction are weighted and added by using the absolute value of the luminance gradient direction signal as a gain, thereby obtaining an image of the image. It is possible to correct defective pixels while minimizing edge degradation.

  Hereinafter, similarly, when the luminance gradient direction signal is negative with the flag High, weighted addition of the corrected image signal in the vertical direction and the corrected image signal in the oblique 135 degree direction, and the luminance gradient direction signal is positive with the flag Low. In the case of the value, the weighted addition of the corrected image signal in the horizontal direction and the corrected image signal in the oblique 45 degree direction, and when the luminance gradient direction signal is negative with the flag Low, the corrected image signal in the vertical direction and the inclined 45 degree direction By performing the weighted addition of the corrected image signals, it is possible to perform the best defective pixel correction.

  As described above, when the defective pixel correction of the G pixel in which the image quality deterioration due to the defective pixel erroneous detection is conspicuous is performed as in the present embodiment, an image with higher image quality can be obtained.

  As described above, according to the above-described embodiment, even if a defective pixel is included in a predetermined range of pixels used when detecting the edge direction, a decrease in edge direction detection accuracy due to the defective pixel is prevented. Therefore, it is not necessary to previously store the defective pixel position in the nonvolatile memory. Accordingly, the process itself for writing the position information of the defective pixel in the nonvolatile memory can be abolished, and the productivity of the imaging apparatus itself is expected to be dramatically improved.

  Further, in the defective pixel correction using the nonvolatile memory, it is necessary to cope with a defective pixel newly generated after the image pickup device is manufactured. Conventionally, for example, Japanese Patent Application Laid-Open Nos. 2001-218115 and 2001-257939. However, according to the above-described embodiment, a defective pixel newly generated after manufacturing can be corrected without any problem.

  Further, since defective pixels are detected along the edge direction of the subject and the defective pixels are corrected along the edge direction of the subject, extremely good defective pixels are generated without causing erroneous detection of high-frequency components such as the subject edge. Correction can be realized.

  The present invention can be applied to all image pickup apparatuses using an image pickup device, such as a digital camera, a video camera, and a camera-equipped mobile phone.

It is a figure which shows the defective pixel correction apparatus of the 1st Embodiment of this invention. FIG. 2 is a detailed view of each defective pixel provisional detection unit in FIG. 1. FIG. 2 is a detailed diagram of a luminance gradient direction detection unit in FIG. 1. FIG. 2 is a detailed view of a two-dimensional defective pixel detection / correction unit in FIG. 1. FIG. 2 is a theoretical explanatory diagram of luminance gradient direction detection processing in FIG. 1. It is a figure which shows the defective pixel correction apparatus of the 2nd Embodiment of this invention. FIG. 7 is a detailed diagram of a luminance gradient direction detection unit orthogonal to the edge direction of the image of FIG. 6. FIG. 8 is a detailed view of each correction block in FIG. 7. FIG. 7 is a detailed diagram of a two-dimensional defective pixel detection / correction unit in FIG. 6. FIG. 10 is a detailed diagram of each detection / correction circuit in FIG. 9. It is explanatory drawing of the mix control of each correction | amendment image signal of the signal mix part of FIG.

Explanation of symbols

1-1 Image signal input terminal 1-21 1st line memory 1-22 2nd line memory 1-23 3rd line memory 1-24 4th line memory 1-31 1st defective pixel temporary detection part 1-32 Second defective pixel provisional detection unit 1-33 Third defective pixel provisional detection unit 1-34 Fourth defective pixel provisional detection unit 1-35 Fifth defective pixel provisional detection unit 1-4 First Threshold input terminal 1-5 Luminance gradient direction detection unit 1-6 Two-dimensional defective pixel detection / correction unit 1-7 Second threshold input terminal 1-8 Changeover switch 1-9 Corrected image output terminal 6-1 Image signal input Terminal 6-21 1st line memory 6-22 2nd line memory 6-23 3rd line memory 6-24 4th line memory 6-31 1st defective pixel temporary detection part 6-32 2nd Defective Pixel Temporary Detection Unit 6-34 Third Defect Image Temporary detection unit 6-35 Fourth defective pixel temporary detection unit 6-4 First threshold input terminal 6-5 Defective pixel correction unit for R and B pixels 6-6 Luminance gradient direction detection unit 6-7 Two-dimensional defect Pixel Detection / Correction Unit 6-8 Second Threshold Input Terminal 6-9 Signal Mix Unit 6-10 G Pixel and RB Pixel Changeover Switch 6-11 G Pixel and RB Pixel Identification Signal Input 6-12 Corrected Image Output Terminal

Claims (3)

A defect pixel detection means for detecting a defective pixel information of an image pickup device having a plurality of pixels to acquire an image signal by imaging an object image,
An image signal obtained from the image sensor, on the basis of the detection result of the defect pixel detector, the edge direction detection means for detecting an edge direction of the object,
Defective pixel detection and correction means for detecting defective pixel information of the image sensor, performing correction of output signals of the detected defective pixels, respectively, using signals of pixels in a plurality of directions, and generating a plurality of correction signals;
Selection means for selecting one correction signal from the plurality of correction signals based on the detection result of the edge direction detection means;
A defective pixel correction apparatus comprising: A plurality of pixels of the image sensor are arranged two-dimensionally, the defect pixel detection means for one-dimensional direction of an image signal obtained from the image sensor, and detects the defective pixel The defective pixel correction apparatus according to claim 1 . A defect pixel detection step for detecting a defective pixel information of an image pickup device having a plurality of pixels to acquire an image signal by imaging an object image,
An image signal obtained from the image sensor, on the basis of the detection result of the defect pixel detection step, the edge direction detection step of detecting an edge direction of the object,
Detecting defective pixel information of the image sensor, performing a correction of the output signal of the detected defective pixel, respectively using a pixel signal in a plurality of directions, and generating a plurality of correction signals, a defective pixel detection correction step,
A selection step of selecting one correction signal from the plurality of correction signals based on the detection result of the edge direction detection step;
A defective pixel correction method comprising:

Images