JP6019587B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus.

  Conventionally, image sensors such as CCD (Charge Coupled Device Image Sensor) and CMOS (Complementary Metal Oxide Semiconductor) incorporated in digital cameras and the like have multiple color filters on the front surface of the plurality of light receiving elements arranged on the light receiving surface. A Bayer array color filter array is provided. However, incident light that has passed through a color filter corresponding to a certain light receiving element may leak into an adjacent light receiving element, and a line crawl is superimposed on the captured image.

  Therefore, for example, a line crawl is removed by extracting a high frequency component from the image and subtracting the high frequency component contained in the luminance signal of the image in accordance with a comparison between the absolute value of the high frequency component and a set threshold value. Technology is disclosed (see Patent Document 1)

Japanese Patent No. 3664636

  However, in the related art, when the image signal of the subject is smaller than the set threshold value, it is determined as a line crawl signal and is removed, so that the fine image structure of the subject is lost.

  SUMMARY OF THE INVENTION In view of the above-described problems of the prior art, an object of the present invention is to provide a technique that can remove line crawls with high accuracy without losing a fine image structure of a subject.

One aspect of the image processing apparatus illustrating the present invention is a first difference value that is a difference between a pixel value of a first pixel and a pixel value of a second pixel in which the arrangement of color components of adjacent pixels is different from the first pixel. A first calculation unit that calculates the second difference value that is the difference between the pixel value of the first pixel and the pixel value of the third pixel in which the arrangement of the color components of the adjacent pixels is the same as the first pixel A second calculator that calculates a third difference that is a difference between the first difference value and the second difference value, and a fourth that subtracts the third difference value from the pixel value of the first pixel. And an arithmetic unit .

Further, the color components of the first pixel and the color component of the second pixel may me identical der.

The arrangement of the color component of the first pixel, the color component of the second pixel, the color component of the pixel adjacent to the first pixel, and the color component of the pixel adjacent to the second pixel may be a Bayer arrangement .

Also, the color component of the first pixel and the color component of the second pixel are green, and the first pixel and the pixel having a red color component among the pixels adjacent to the first pixel are in the same row of the Bayer array. The second pixel and the pixel whose color component is blue among the pixels adjacent to the second pixel may be arranged in the same row of the Bayer array .

In addition, the color component of the first pixel and the color component of the second pixel are green, and the first pixel and the pixel having the blue color component among the pixels adjacent to the first pixel are in the same row of the Bayer array. The second pixel and the pixel whose color component is red among the pixels adjacent to the second pixel may be arranged in the same row of the Bayer array .

  According to the present invention, line crawl can be removed with high accuracy without losing the fine image structure of the subject.

The figure which shows the structure of the digital camera which concerns on one Embodiment of this invention. The figure which shows an example of the Bayer arrangement | sequence in an image sensor 7 is a flowchart showing processing operations of a digital camera according to an embodiment. The figure which shows an example of the processing result according to the step of FIG. The figure which shows an example of the result from which the line crawl component was removed The figure which shows the structure of the computer which concerns on other embodiment of this invention. The flowchart which shows the processing operation of the computer which concerns on other embodiment.

<< One Embodiment >>
FIG. 1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention.

  The digital camera of this embodiment includes an imaging optical system 11, an imaging device 12, an AFE 13, a buffer memory 14, an image processing unit 15, a monitor 16, a storage unit 17, a CPU 18, an operation member 23, a recording I / F 24, and a bus. The The buffer memory 14, the image processing unit 15, the monitor 16, the storage unit 17, the CPU 18, and the recording I / F 24 are respectively connected via a bus so that information can be transmitted. The operation member 23 is connected to the CPU 18.

  The imaging optical system 11 includes a plurality of lens groups including a zoom lens and a focusing lens. The lens position of the imaging optical system 11 is adjusted in the optical axis direction by a lens driving unit (not shown). For simplicity, the imaging optical system 11 is illustrated as a single lens in FIG.

  The image pickup device 12 is a device that picks up an image of a subject imaged by a light beam that has passed through the image pickup optical system 11. The output of the image sensor 12 is input to the AFE 13. Note that the image sensor 12 of the present embodiment may be a progressive scan type solid-state image sensor (CCD or the like) or an XY address type solid-state image sensor (CMOS or the like).

  The image pickup device 12 picks up a recorded image (main image) in response to a full press operation of a release button constituting the operation member 23 in the shooting mode of the digital camera. Further, the imaging element 12 in the shooting mode captures a composition confirmation image (through image) at a predetermined time interval even during shooting standby. The through image data is output from the image sensor 12 by thinning-out readout. The through image data is used for image display on the monitor 16 and various arithmetic processes by the CPU 18.

  Further, as shown in FIG. 2, a plurality of light receiving elements are arranged in a matrix on the light receiving surface of the image pickup element 12. In each light receiving element of the image sensor 12, red (R), green (Gr, Gb), and blue (B) color filters are arranged according to a known Bayer array. Therefore, each light receiving element of the imaging element 12 outputs an image signal corresponding to each color by color separation in the color filter. Thereby, the image sensor 12 can acquire a color image. FIG. 2 shows an arrangement of some light receiving elements and color filters of the image sensor 12.

  Here, the line crawl will be briefly described. For example, when red light is incident on the light receiving surface of the image sensor 12 obliquely from the subject, the red light is usually only a light receiving element (hereinafter referred to as an R pixel) in which an R filter that transmits red is disposed. Is received. However, part of the light transmitted through the R filter leaks into a light receiving element (hereinafter referred to as a Gr pixel or a Gb pixel) in which an adjacent Gr or Gb filter is arranged depending on the incident direction. As a result, the Gr pixel or Gb pixel affected by leakage has a different pixel value from the Gb pixel or Gr pixel that does not leak, and the line has different characteristics for each row or column in the so-called horizontal or vertical scanning direction. Crawls overlap. The same applies to the case where blue light is incident obliquely. In the following, G pixels affected by leakage will be referred to as a first pixel group, and G pixels without leakage will be referred to as a second pixel group.

  The AFE 13 is an analog front end circuit that performs analog signal processing on the output of the image sensor 12. The AFE 13 performs correlated double sampling, image signal gain adjustment, and image signal A / D conversion. The output of the AFE 13 is temporarily recorded in the buffer memory 14. In the present embodiment, the imaging device 12 and the AFE 13 constitute an imaging unit. The buffer memory 14 can be used by appropriately selecting a general volatile semiconductor memory or the like, and temporarily stores image data in the pre-process and post-process of the image processing by the image processing unit 15. .

  The image processing unit 15 performs various types of image processing (for example, color interpolation processing, gradation conversion processing, white balance processing, contour enhancement processing) on the digital image signal for one frame stored in the buffer memory 14. .

  The monitor 16 is a display unit such as a liquid crystal monitor, and displays various images in accordance with instructions from the CPU 18.

  The storage unit 17 stores data of a main image captured by a digital camera, a control program executed by the CPU 18, an image processing program, and the like. Note that a nonvolatile semiconductor memory or the like can be used for the storage unit 17.

  The CPU 18 is a processor that comprehensively controls each unit of the digital camera. By executing the control program and the image processing program, the CPU 18 performs known automatic focus (AF) control by the phase difference detection method or contrast detection method, automatic exposure (AE) calculation, and the like based on the data of the through image. Do. Further, the CPU 18 performs a file generation process for the captured image, a display process for the monitor 16, and the like. Furthermore, the CPU 18 of this embodiment operates as a first calculation unit 19, a second calculation unit 20, a correction value acquisition unit 21, and a removal unit 22 by executing the image processing program.

The first calculation unit 19 uses the following equation (1) for the pixel value of the target pixel for the line crawl removal process to correct the pixel value of the first pixel group affected by the leakage, A first high-frequency component E1 (i, j) included in the pixel value of the target pixel is calculated from the pixel values of the target pixel and the pixels of the adjacent second pixel group. In the following description, the pixel of the first pixel group is represented as G1, the pixel of the second pixel group is represented as G2, and when the target pixel to be removed is Gr, G1 = Gr and G2 = Gb. in the case of G b becomes G1 = Gb, G2 = Gr.
E1 (i, j) = [G1 (i, j)-(G2 (i-1, j-1) + G2 (i-1, j + 1) + G2 (i + 1, j-1) + G2 (i + 1, j + 1)) / 4] / 2 (1)
The second calculation unit 20 calculates the second high-frequency component E2 (i, j) included in the pixel value of the target pixel from the pixel value of the target pixel and the adjacent first pixel group using the following equation (2). To do.
E2 (i, j) = [G1 (i, j)-(G1 (i-2, j-2) + G1 (i-2, j) + G1 (i-2, j + 2) + G1 (i, j-2)] + G1 (i, j + 2) + G1 (i + 2, j-2) + G1 (i + 2, j) + G1 (i + 2, j + 2)) / 8] / 2 (2)
Since the second high frequency component E2 calculated by the second calculation unit 20 is obtained only from the first pixel group, it indicates the high frequency component of the local image structure at the pixel position (i, j) of the target pixel. . On the other hand, since the first high-frequency component E1 calculated by the first calculation unit 19 is obtained from the first pixel group and the second pixel group, it has a local image structure at the pixel position (ij) of the target pixel. The high frequency component containing a line crawl component is shown.

The correction value acquisition unit 21 uses the following equation (3) to subtract the second high-frequency component E2 from the first high-frequency component E1 in the target pixel, and clip the value between 0 and E1 (i, j) To do. Thereby, the correction value acquisition unit 21 can extract only the line crawl component L (i, j) by removing the structural component of the subject as described above.
L (i, j) = clip [0, E1 (i, j)] (E1 (i, j) −E2 (i, j)) (3)
In the present embodiment, the line crawl component L (i, j) is used as the line crawl correction value for the target pixel G1 (i, j).

  The removing unit 22 removes the line crawl by subtracting the line crawl component L (i, j) from the pixel value of the target pixel G1 (i, j).

  The operation member 23 includes, for example, a release button, a command dial, a cross-shaped cursor key, a determination button, and the like. The operation member 23 receives various inputs of the digital camera from the user. For example, the operation member 23 is used for an imaging operation, a digital camera operation mode switching operation, an input operation on a setting screen, and the like.

  A connector for connecting the storage medium 25 is formed in the recording I / F 24. The recording I / F 24 writes / reads data to / from the storage medium 25 connected to the connector. The storage medium 25 is composed of a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is shown as an example of the storage medium 25.

  Next, the processing operation by the digital camera of the present embodiment will be described with reference to the flowchart of FIG. 3 and FIG. 4 showing the processing result of each step. 4A to 4D are diagrams in which part of pixel values of Gr pixels and Gb pixels on two adjacent horizontal scanning lines in the image are arranged on the horizontal axis in the horizontal scanning direction. The pixel value distribution of the G pixel before processing, the distribution of the first high-frequency component E1, the distribution of the second high-frequency component E2, and the distribution of the line crawl component L are respectively shown. For example, in FIG. 4A, the wavy distribution of pixel values indicates the influence of line crawl, and the steps with large pixel values indicate the edge structure of the subject.

  In the following, the case where the digital camera of the present embodiment captures a subject as a single image and obtains a RAW image of a still image will be described as an image capture mode. However, the same applies to image capture modes such as continuous shooting and moving images. Detailed description will be omitted.

  The CPU 18 accepts a power-on instruction by turning on the power button on the operation member 23 by the user and turns on the digital camera. The CPU 18 reads and executes the control program and the image processing program from the storage unit 17 and initializes the digital camera. The CPU 18 stands by until an instruction for imaging the subject from the user is issued.

  Step S101: The CPU 18 determines whether or not the user has fully pressed the release button of the operation member 23. When the CPU 18 receives an operation of fully pressing the release button of the operation member 23, the CPU 18 determines that an imaging instruction has been issued and issues an imaging command to a timing generator (not shown). The timing generator (not shown) outputs a timing pulse to the image sensor 12, and the image sensor 12 images the subject imaged by the imaging optical system 11. The CPU 18 converts the captured RAW image data into digital data by the AFE 13 and temporarily records it in the buffer memory 14. The CPU 18 proceeds to step S102.

  On the other hand, if the CPU 18 has not received a full press operation, the CPU 18 waits until a full press operation is received.

  Step S102: The first calculation unit 19 of the CPU 18 reads the RAW image data from the buffer memory 14, and calculates the first high-frequency component E1 (i, j) of the pixel value in the target pixel based on Expression (1) ( FIG. 4 (b)).

  Step S103: The second calculation unit 20 of the CPU 18 calculates a second high-frequency component E2 (i, j) of the pixel value in the target pixel of the RAW image data based on the equation (2). As shown in FIG. 4C, since the second high frequency component E2 of the pixels other than the pixel corresponding to the image structure of the subject is 0 as compared with FIG. 4B, the second high frequency component E2 is It can be seen that this component reflects the image structure.

  Step S104: The correction value acquisition unit 21 of the CPU 18 calculates the line crawl component L (i, j) by subtracting the second high frequency component E2 from the first high frequency component E1 based on the equation (3). The crawl correction value. As shown in FIG. 4D, the image structure of the subject is removed by the subtraction, and only the line crawl component L (i, j) is extracted.

  Step S105: The CPU 18 determines whether or not the line crawl component L (i, j) has been acquired for all the Gr and Gb pixels. If the CPU 18 has acquired all the Gr and Gb pixels, the CPU 18 proceeds to step S106 (YES side). On the other hand, if the CPU 18 has not acquired the line crawl component L (i, j) for all the Gr and Gb pixels, the CPU 18 proceeds to step S102 (NO side). The CPU 18 performs the processing from step S102 to step S104 until the line crawl component L of all Gr and Gb pixels is acquired.

  Step S106: The removal unit 22 of the CPU 18 subtracts the line crawl component L (i, j) from the pixel value of the target pixel (i, j) to remove the line crawl. FIG. 5A shows the result of subtracting the line crawl component L (i, j) of FIG. 4D from FIG. FIG. 5 (b) also shows a result of subtracting the first high frequency component E1 of FIG. 4 (b) from FIG. 4 (a), which has been conventionally performed, as a reference. Obviously, the image structure of the subject is retained in FIG. 5A, and the image structure is dulled in FIG. 5B.

  Step S107: The CPU 18 outputs the RAW image from which the line crawl has been removed to the image processing unit 15. The image processing unit 15 performs various image processing such as color interpolation processing, gradation conversion processing, white balance processing, and contour enhancement processing on the RAW image, and outputs a still image.

  Step S108: The CPU 18 generates an image file of the still image processed in step S107 and records it in the storage unit 17 and the storage medium 25.

  Note that the CPU 18 of the present embodiment may record the raw image data in the storage unit 17 or the storage medium 25 in association with the still image together with the still image file. Further, when generating an image file of a still image, the CPU 18 preferably adds information such as exposure conditions and metadata conforming to the Exif (Exchangeable image file format for digital still cameras) standard to the header area of the image file. As a result, the line crawl superimposed on the RAW image can be removed even in image processing by a computer to be described later.

Thus, in this embodiment, the two first high-frequency components E1 and the second high-frequency components E2 are calculated, and the line crawl correction amount is acquired based on them, thereby eliminating the fine image structure of the subject. It is possible to remove the line crawl with high accuracy without causing it to occur.
<< Other embodiments >>
FIG. 6 is a block diagram of a computer 100 that operates as an image processing apparatus by executing an image processing program according to another embodiment of the present invention.

  A computer 100 shown in FIG. 6A includes a CPU 50, a storage unit 51, an input / output interface (input / output I / F) 52, and a bus 53, and the CPU 50, storage unit 51, and input / output I / F 52 include a bus 53. It is connected so that information can be transmitted through the network. The computer 100 is connected to an output device 60 for displaying the progress of image processing and processing results, and an input device 70 for receiving input from the user, via the input / output I / F 52. For the output device 60, a general liquid crystal monitor, a printer, or the like can be selected and used for the input device 70, such as a keyboard or a mouse, respectively.

  The CPU 50 is a microprocessor that comprehensively controls each unit of the computer 100. For example, the CPU 50 reads an image processing program stored in the storage unit 51 and performs image processing on the image data stored in the storage unit 51 based on an instruction from the user received by the input device 70. The CPU 50 displays the image processing result on the output device 60. As shown in FIG. 1B, the CPU 50 according to the present embodiment executes the image processing program as in the case of the CPU 18 of the digital camera according to the first embodiment. The correction value acquisition unit 21 and the removal unit 22 operate. In addition, since the 1st calculation part 19, the 2nd calculation part 20, the correction value acquisition part 21, and the removal part 22 of this embodiment operate | move similarly to the thing of one embodiment, it is the thing of one embodiment. The same reference numerals are used and detailed description is omitted.

  The storage unit 51 stores an image file of a subject imaged by the digital camera of one embodiment and its RAW image data. Note that these image files and RAW image data are acquired from the digital camera of one embodiment, directly or via a network by wire or wireless, or acquired via the storage medium 25 and stored. Shall.

  The storage unit 51 also records an image processing program that removes line crawls for the image. For the storage unit 51, a general storage device such as a hard disk device or a magneto-optical disk device can be selected and used. 6 is incorporated in the computer 100, it may be an external storage device. In this case, the storage unit 51 is connected to the computer 100 via the input / output I / F 52.

  Next, the processing operation by the computer 100 of this embodiment will be described with reference to the flowchart of FIG. In the present embodiment, it is assumed that the raw image data acquired in advance by the above means is stored in the storage unit 51.

  The user inputs an image processing program command using the input device 70 or double-clicks the program icon to issue a program start command. The CPU 50 receives the command via the input / output I / F 52 and executes the image processing program stored in the storage unit 51. As a result, the processing from step S201 in FIG. 7 is performed.

  In step S <b> 201, the CPU 50 acquires and reads the file name of the RAW image data designated by the user via the input device 70. And CPU50 performs the process of step S202-step S208. Note that step S202 to step S208 are the same as step S102 to step S108 of the embodiment, and thus detailed description thereof is omitted.

Thus, in this embodiment, the two first high-frequency components E1 and the second high-frequency components E2 are calculated, and the line crawl correction amount is acquired based on them, thereby eliminating the fine image structure of the subject. It is possible to remove the line crawl with high accuracy without causing it to occur.
<< Additional items of embodiment >>
(1) In the above embodiment, line crawls are removed from all RAW image data, but the present invention is not limited to this. For example, the line crawl may not be noticeable depending on the imaging environment (for example, the imaging condition such as the F value), and thus the CPU 18 and the CPU 50 perform the line crawl removal process according to the imaging environment in which the RAW image is captured. Whether or not line crawl may be removed according to the determination result. In addition, since the light from the subject is likely to be incident obliquely in the edge image area of the RAW image, the CPU 18 and the CPU 50 may perform the line crawl removal process only on the pixels in the edge image area. Good. Alternatively, the CPU 18 and the CPU 50 may perform color distribution in the RAW image data and scene recognition of the subject, and may perform line crawl removal processing in the case of a subject with many red and blue colors.

(2) In the above embodiment, the correction value acquisition unit 21 extracts the line crawl component L using the equation (3), but the present invention is not limited to this. For example, the correction value acquisition unit 21 may extract the line crawl component L using the following equation (4) having a predetermined constant k.
L (i, j) = clip [0, E1 (i, j)] (E1 (i, j) −k × E2 (i, j)) (4)
Here, the reason for using the predetermined constant k is that the first high-frequency component E1 is obtained from the pixel values of the target pixel G1 and the adjacent pixel G2, and therefore the line crawl component is detected rather than the image structure of the subject. easy. On the other hand, since the second high-frequency component E2 is obtained from the pixel values of the target pixel G1 and the adjacent pixel G1, the image structure of the subject is detected. Therefore, by adjusting the value of k according to the imaging environment or the like, for example, if k is larger than 1, loss of the image structure of the subject can be suppressed, and if k is smaller than 1, line crawl can be strongly suppressed. .

Moreover, the correction value acquisition part 21 may extract the line crawl component L using following Formula (5).
U = k ′ × | E2 (i, j) | − | E1 (i, j) |
S = E1 (i, j) / | E1 (i, j) |
L (i, j) = clip [0, E1 (i, j)] (E1 (i, j) −S × U) (5)
Here, k ′ is a predetermined constant, set to about 1, and U is clipped to a value of 0 or more. S indicates the sign of the first high-frequency component E1. By using this equation (5), the line crawl correction is suppressed only when the second high frequency component E2 of the RAW image data is larger than the first high frequency component E1 including the line crawl component, and compared with the equation (3). Thus, line crawl is strongly suppressed in a region where the image structure of the subject is flat.

  (3) In the above one embodiment, the example in which the CPU 18 implements each process of the first calculation unit 19, the second calculation unit 20, the correction value acquisition unit 21, and the removal unit 22 in software has been described. Each of these processes may be realized by hardware using an ASIC. In one embodiment, line crawl removal processing is performed before Bayer interpolation. However, when ASIC is used, it is preferable to incorporate this line crawl removal processing in the processing for generating the luminance of Bayer interpolation. That is, when designing a circuit by pipeline processing, it is possible to set the peripheral pixels to be referred to by Bayer interpolation and the peripheral pixels to be referred to by line crawl removal to be read from a common line memory. Can be reduced.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Imaging optical system, 12 ... Imaging element, 13 ... AFE, 14 ... Buffer memory, 15 ... Image processing part, 16 ... Monitor, 17, 51 ... Memory | storage part, 18, 50 ... CPU, 19 ... 1st calculation part, DESCRIPTION OF SYMBOLS 20 ... 2nd calculation part, 21 ... Correction value acquisition part, 22 ... Removal part, 23 ... Operation member, 24 ... Recording I / F, 25 ... Storage medium, 53 ... Input / output I / F, 54 ... Bus, 60 ... Output device 70 ... Input device 100 ... Computer

Claims (5)

A first calculation unit that calculates a first difference value that is a difference between a pixel value of a first pixel and a pixel value of a second pixel in which the arrangement of color components of adjacent pixels is different from the first pixel;
A second calculation unit that calculates a second difference value that is a difference between a pixel value of the first pixel and a pixel value of a third pixel in which the arrangement of the color components of adjacent pixels is the same as the first pixel;
A third calculation unit that calculates a third difference value that is a difference between the first difference value and the second difference value;
An image processing apparatus comprising: a fourth calculation unit that subtracts the third difference value from a pixel value of the first pixel. The image processing apparatus according to claim 1.
The image processing apparatus, wherein the color component of the first pixel and the color component of the second pixel are the same. The image processing apparatus according to claim 2,
The image processing apparatus according to claim 1, wherein an array of the color component of the first pixel, the color component of the second pixel, the color component of the pixel adjacent to the first pixel, and the color component of the pixel adjacent to the second pixel is a Bayer array. The image processing apparatus according to claim 3.
The color component of the first pixel and the color component of the second pixel are green,
The first pixel and the pixel having a red color component among the pixels adjacent to the first pixel are arranged in the same row of the Bayer arrangement,
The image processing apparatus in which the second pixel and the pixel having a blue color component among the pixels adjacent to the second pixel are arranged in the same row of the Bayer array. The image processing apparatus according to claim 3.
The color component of the first pixel and the color component of the second pixel are green,
The first pixel and the pixel whose color component is blue among the pixels adjacent to the first pixel are arranged in the same row of the Bayer arrangement,
The image processing apparatus in which the second pixel and the pixel having a red color component among the pixels adjacent to the second pixel are arranged in the same row of the Bayer array.

Images