JP6198664B2 - Image processing method, image processing apparatus, imaging apparatus, image processing program, and storage medium - Google Patents

Description

  The present invention relates to an image processing method for improving the quality of a captured image.

  When imaging is performed by an imaging device such as a camera, part of light incident on the imaging optical system may be reflected by the lens interface or a member holding the lens and reach the imaging surface as unnecessary light. Unnecessary light reaching the imaging surface appears in the captured image as unnecessary components such as ghosts and flares. In addition, if a diffractive optical element is used to correct axial chromatic aberration or lateral chromatic aberration, light from a high-intensity object such as the sun outside the field of view strikes the diffractive optical element, which is unnecessary for the entire image. Light may appear as an unwanted component.

  Patent Document 1 describes the difference between an image when the imaging optical system is in focus with respect to a subject (focused image) and an image when the imaging optical system is out of focus (defocused image). A method for detecting a ghost from the difference image shown is disclosed. However, the method of Patent Document 1 requires a plurality of times of imaging, and is not suitable for still image imaging or moving image imaging of a moving subject.

  Patent Document 2 discloses a method for detecting a ghost by comparing a plurality of viewpoint images obtained by monocular stereoscopic imaging. In the method of Patent Document 2, since a plurality of parallax images can be obtained by one imaging, it is possible to cope with still image imaging and moving image imaging of a moving subject.

JP 2008-54206 A JP 2011-205531 A

  However, since the ghost is detected by obtaining the difference between the two images of the main image and the sub-image in the method of Patent Document 2, the ghost detection effect is reduced in the case of viewpoint images with three or more viewpoints. In Patent Document 2, image shift due to parallax is corrected before ghost detection. Focused subjects can be dealt with by extracting corresponding points. However, it is not enough to extract the corresponding points for blurred subject areas, and detailed distance information (distance map) from the imaging device to the subject throughout the screen. ) Need to get. On the other hand, if a ghost is detected without correcting the parallax deviation, a parallax component is also detected together with the ghost, and the parallax component and the unnecessary component (ghost component) cannot be separated.

  The present invention provides an image processing method, an image processing apparatus, an imaging apparatus, an image processing program, and a storage medium that can separate a parallax component and an unnecessary component included in a captured image without performing a plurality of imaging.

An image processing method according to an aspect of the present invention includes: determining a plurality of first unnecessary components of each of the plurality of parallax images based on a plurality of pieces of relative difference information of the plurality of parallax images; and a step of determining determining direction component information, the second unnecessary components based on previous SL plurality of first unnecessary component and the direction component information based on the unnecessary components.

An image processing apparatus according to another aspect of the present invention includes a first unnecessary component determination unit that determines a plurality of first unnecessary components related to each of the plurality of parallax images based on a plurality of pieces of relative difference information of the plurality of parallax images. second unnecessary components determining the directional component determining means for determining the directional component information, the second unnecessary components based on previous SL plurality of first unnecessary component and the direction component information based on the plurality of first unnecessary components Determination means.

An imaging apparatus as another aspect of the present invention includes an imaging element that photoelectrically converts an optical image and outputs a plurality of parallax images, and the image processing apparatus .

An image processing program according to another aspect of the present invention includes a step of determining a plurality of first unnecessary components related to each of the plurality of parallax images based on a plurality of pieces of relative difference information of the plurality of parallax images; It is configured to execute determining a direction component information in accordance with an unnecessary component, determining a second required component on the basis of the previous SL plurality of first unnecessary component and the direction component information, to the computer ing.

  A storage medium according to another aspect of the present invention stores the image processing program.

  Other objects and features of the present invention are illustrated in the following examples.

  According to the present invention, an image processing method, an image processing apparatus, an imaging apparatus, an image processing program, and a storage medium that can separate a parallax component and an unnecessary component included in a captured image without performing a plurality of imaging are provided. be able to.

In the imaging system of each Example, it is a related figure of the light-receiving part of an image sensor, and the pupil of an imaging optical system. It is a schematic diagram of the imaging system in each Example. It is a schematic diagram of the pupil of the imaging system in each embodiment. It is a block diagram of the imaging device in each embodiment. 2 is an explanatory diagram of a configuration of an imaging optical system and unnecessary light generated in the imaging optical system in Embodiment 1. FIG. 6 is an explanatory diagram of unnecessary light that passes through a diaphragm of an imaging optical system in Embodiment 1. FIG. In Example 1, it is a figure which shows the picked-up image produced | generated by the imaging which does not perform pupil division. 6 is a diagram illustrating a parallax image in Embodiment 1. FIG. FIG. 3 is a diagram illustrating a procedure of an image processing method in Embodiment 1. FIG. 3 is a diagram illustrating a procedure of an image processing method in Embodiment 1. FIG. 3 is a diagram illustrating a procedure of an image processing method in Embodiment 1. FIG. 3 is a diagram illustrating a procedure of an image processing method in Embodiment 1. FIG. 6 is a diagram illustrating an example of an output image obtained by the image processing method according to the first embodiment. 3 is a flowchart illustrating an image processing method according to the first exemplary embodiment. FIG. 3 is a diagram illustrating a procedure of an image processing method in Embodiment 1. FIG. 6 is a diagram illustrating a light receiving unit of an image sensor in Example 2. FIG. 6 is an explanatory diagram of unnecessary light that passes through a diaphragm of an imaging optical system in Embodiment 2. FIG. 6 is a diagram illustrating a light receiving unit of an image sensor in Example 2. FIG. 10 is a diagram illustrating a procedure of an image processing method in Embodiment 2. 6 is a diagram illustrating an imaging system in Embodiment 3. FIG. 6 is a diagram illustrating an imaging system in Embodiment 3. FIG. 6 is a diagram illustrating an imaging system in Embodiment 3. FIG. It is a figure which shows the conventional image pick-up element. It is a figure which shows the image obtained by the imaging system of FIG. It is a figure which shows the image obtained by the imaging system of FIG. 21 and FIG. FIG. 10 is a diagram illustrating an example of an imaging device according to a third embodiment. FIG. 10 is a diagram illustrating an example of an imaging device according to a third embodiment. FIG. 10 is an explanatory diagram of a region where unnecessary component reduction processing is performed in Example 4;

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The imaging apparatus capable of generating a plurality of parallax images used in the present embodiment guides a plurality of light beams that have passed through different regions of the pupil of the imaging optical system to different light receiving units (pixels) in the imaging device to perform photoelectric conversion. It has an imaging system that performs.

  FIG. 1 is a diagram illustrating a relationship between a light receiving unit of an imaging element and a pupil of an imaging optical system in the imaging system of the present embodiment. In FIG. 1, ML is a microlens, and CF is a color filter. EXP is an exit pupil (pupil) of the imaging optical system, and P1 and P2 are areas of the exit pupil EXP. G1, G2, G3, and G4 are pixels (light receiving portions), and the pixels G1 to G4 constitute one set of pixels (pixel set). A plurality of pixel groups of pixels G1 to G4 are arranged in the imaging element. The pixel G3 is two-dimensionally arranged on the back side of the paper surface of the pixel G1, and the pixel G4 is two-dimensionally arranged on the back side of the paper surface of the pixel G2. The pixels G1 to G4 have a conjugate relationship with the exit pupil EXP via a common microlens ML (that is, one for each pixel set). In the present embodiment, the plurality of pixels G1 to G4 arranged in the image sensor may be collectively referred to as a pixel group G1 to G4, respectively.

  FIG. 2 shows an imaging relationship among the object point OSP being imaged, the exit pupil EXP, and the image sensor. The pixel G1 receives the light beam that has passed through the region P1 in the exit pupil EXP. The pixel G2 receives the light beam that has passed through the region P2 in the exit pupil EXP. The region P3 exists two-dimensionally on the back side of the paper surface of the region P1, and the region P4 exists two-dimensionally on the back side of the paper surface of the region P2. The object point OSP does not necessarily have an object. The light beam that has passed through the object point OSP is incident on one of the pixels G1 to G4 according to the position (region P1 to P4 in this embodiment) in the pupil (exit pupil EXP) through which the light beam passes. . The passage of light beams through different regions in the pupil corresponds to the separation of incident light from the object point OSP by the angle (parallax). That is, among the pixels G1 to G4 provided for each microlens ML, four images generated using the output signals from the pixels G1 to G4 become a plurality of parallax images having parallax. In the following description, receiving light beams that have passed through different regions in the pupil by different light receiving units (pixels) may be referred to as pupil division. FIG. 3 is a view of the exit pupil EXP as viewed from the microlens ML side.

  Further, even when the above-described conjugate relationship is not perfect due to the positional deviation of the exit pupil EXP shown in FIGS. 2 and 3, or when the regions P1 and P2 partially overlap (overlap), for example, In each of the examples, a plurality of obtained images are handled as parallax images.

  With reference to FIG. 4, an imaging apparatus that executes the image processing method according to the first embodiment of the present invention will be described. FIG. 4 is a block diagram illustrating a configuration of the imaging apparatus 200 in the present embodiment. The imaging optical system 201 includes a diaphragm 201a and a focus lens 201b, and focuses (condenses) light from a subject (not shown) on the imaging element 202. The imaging element 202 is configured by a photoelectric conversion element such as a CCD sensor or a CMOS sensor, and a light beam that has passed through different areas in the pupil described with reference to FIG. 1 and FIG. Part) to receive light (perform pupil division). In this way, the image sensor 202 photoelectrically converts the subject image (optical image) and outputs image signals (analog electrical signals) as a plurality of parallax images. The A / D converter 203 converts the analog electric signal output from the image sensor 202 into a digital signal, and outputs the digital signal to the image processing unit 204.

  The image processing unit 204 performs image processing that is generally performed on the digital signal, and also performs unnecessary light determination processing and correction processing that reduces or eliminates unnecessary light. In this embodiment, the image processing unit 204 corresponds to an image processing device mounted on the imaging device 200. The image processing unit 204 includes a first unnecessary component detection unit 204a, a direction component generation unit 204b, a mask creation unit 204c, a second unnecessary component detection unit 204d, and an unnecessary component reduction unit 204e.

  The first unnecessary component detection unit 204a (first unnecessary component determination means) generates (acquires) a plurality of parallax images, and detects (determines) the first unnecessary component from each parallax image. The direction component generation unit 204b (direction component determination means) compares the plurality of detected first unnecessary components, extracts a common luminance component at the same coordinate position in the plurality of first unnecessary components, and obtains the direction component information. decide. Here, “extraction of common luminance component” means a portion having the same luminance value when the luminance values at the same coordinate position of two arbitrary first unnecessary components among the plurality of detected first unnecessary components are compared. The process of extracting only. Or, if a certain difference is allowed and the luminance values are compared, if the luminance difference is ± X%, it has a predetermined luminance difference (threshold value) as a parameter so that it is a common luminance component. A process for extracting a luminance component. The mask creation unit 204c (mask creation means) creates a mask (mask image) based on the direction component. The second unnecessary component detection unit 204d (second unnecessary component determination means) determines the second unnecessary component of each parallax image based on the first unnecessary component and the mask. The unnecessary component reducing unit 204e (unnecessary component reducing means) reduces an unnecessary component (ghost component) from each parallax image based on the second unnecessary component.

  The output image (image data) processed by the image processing unit 204 is stored in an image recording medium 209 such as a semiconductor memory or an optical disk. Further, the output image from the image processing unit 204 can be displayed on the display unit 205. The storage unit 208 stores an image processing program and various information necessary for image processing by the image processing unit 204.

  The system controller 210 (control means) controls the operation of the image sensor 202, the processing in the image processing unit 204, and the imaging optical system 201 (the diaphragm 201a and the focus lens 201b). The imaging optical system control unit 206 mechanically drives the aperture 201a and the focus lens 201b of the imaging optical system 201 in accordance with a control instruction from the system controller 210. The aperture of the aperture 201a is controlled according to the set aperture value (F number). The position of the focus lens 201b is controlled by an auto focus (AF) system (not shown) or a manual focus mechanism in order to perform focus adjustment (focus control) according to the subject distance. The state detection unit 207 acquires current shooting condition information in accordance with a control instruction from the system controller 210. In this embodiment, the image pickup optical system 201 is configured as a part of the image pickup apparatus 200 including the image pickup element 202 (integrated with the image pickup apparatus 200), but is not limited thereto. As with a single-lens reflex camera, an imaging system configured such that an interchangeable imaging optical system (interchangeable lens) is detachable from the imaging apparatus main body may be used.

  FIG. 5 is an explanatory diagram of the configuration of the imaging optical system 201 and unnecessary light generated in the imaging optical system 201. FIG. 5A shows a specific configuration example of the imaging optical system 201. In FIG. 5A, STP is a diaphragm (corresponding to the diaphragm 201a), and IMG is an imaging surface. At the position of the imaging surface IMG, the imaging element 202 shown in FIG. 4 is arranged. FIG. 5B shows that unnecessary light (ghost or flare) is incident on the imaging optical system 201 when strong light is incident on the imaging optical system 201 from the sun SUN as an example of a high-brightness object and is reflected at the lens interface constituting the imaging optical system 201. It shows how the image plane IMG is reached.

  FIG. 6 shows regions P1 to P4 (pupil regions or pupil division regions) through which light beams incident on the pixels G1 to G4 shown in FIG. The stop STP can be considered as corresponding to the exit pupil EXP of the imaging optical system 201, but in reality, the stop STP and the exit pupil EXP are often different from each other. The light beam from the high-intensity object (sun SUN) passes through almost the entire area of the stop STP, but the region through which the light beam incident on the pixels G1 to G4 passes is divided into regions P1 to P4 (pupil regions).

  Next, with reference to FIG. 7 to FIG. 15, a procedure of determination processing (image processing) of an unnecessary component (ghost component) that appears in a captured image due to photoelectric conversion of unnecessary light will be described.

  FIG. 7 shows a captured image Im0 generated by imaging without pupil division. The captured image Im0 in FIG. 7 shows three subjects (subject A, subject B, subject C) and a ghost generated by a light source (not shown). In addition, the captured image Im0 in FIG. 7 is an example of an image in which the subject A is in focus and the subject B and the subject C are blurred. In addition, a ghost shown as a white square portion in the captured image is an unnecessary component. In FIG. 7, the unnecessary component (ghost) is shown painted white, but the background is actually transparent to some extent. In addition, the unnecessary component is a portion in which the luminance of the photographing subject is higher than that of the photographing subject because the unnecessary light is applied to the photographing subject. The same applies to other embodiments described later.

  FIG. 8 is a diagram illustrating a parallax image. FIGS. 8A to 8D show the parallax obtained as a result of photoelectric conversion of the light beams that have passed through the regions P1, P2, P3, and P4 (pupil regions) in the pixel groups G1, G2, G3, and G4, respectively. The conceptual diagram of the images Im1-Im4 is shown. In the case of a short-distance subject, the parallax images Im1 to Im4 include a difference (parallax component) corresponding to the parallax in the image component. When the short-distance subject (subject A) as shown in FIG. 7 is in focus and the rear subject B and subject C are blurred, the subject B and subject C include parallax components. The parallax image Im3 includes an unnecessary component (ghost) schematically shown as a white square. Here, an example of a state where unnecessary components (ghosts) are separated without overlapping each other is shown, but the unnecessary components may overlap each other and have a luminance difference. That is, it is only necessary that the positions and luminances of the white square unnecessary components are different from each other.

  FIG. 9 is a diagram showing the procedure of the image processing method in the present embodiment. 9E to 9P are images (relative difference images Im1-2 to Im4-3) that are differences between the parallax images Im1 to Im4 shown in FIGS. 9A to 9D and the reference image. ). For example, FIG. 9E illustrates a result (relative difference image Im1-2) obtained by subtracting the parallax image Im2 in FIG. 9B from the parallax image Im1 (reference image) in FIG. 9A. FIG. 9F shows a result (relative difference image Im2-1) obtained by subtracting the parallax image Im1 of FIG. 9A from the parallax image Im2 (reference image) of FIG. 9B. FIGS. 9G to 9P also show results obtained according to the same concept. The subtracted image (relative difference image) includes the parallax component of the subject and the aforementioned unnecessary component (ghost) as the difference (relative difference information) of the parallax image. Moreover, although the unnecessary component may be calculated as a negative value by the difference calculation, in order to simplify the unnecessary component reduction process described later, the negative value is rounded down in FIGS. ing. Therefore, an unnecessary component (ghost) included in the parallax image Im3 in FIG. 9C appears only in FIGS. 9G, 9K, and 9O.

  FIGS. 9 (q), (r), (s), and (t) show images in which the relative difference images are compared with each other and the portions having high luminance values are collected and collected. Specifically, FIG. 9 (q) is an image generated by comparing the same coordinates of the relative difference images of FIGS. 9 (e), (i), and (m) and extracting the largest luminance value. MAX1. FIG. 9 (r) is an image MAX2 generated by comparing the same coordinates of the relative difference images of FIGS. 9 (f), 9 (j), and 9 (n) and extracting the largest luminance value. Similarly, FIGS. 9S and 9T are images MAX3 and MAX4 generated based on the relative difference images of FIGS. 9G to 9P. By such image processing, unnecessary components can be left from the parallax images Im1 to Im4 (in other words, unnecessary components can be separated or extracted from the parallax images).

  By the way, in the images of FIGS. 9 (q) to 9 (t), not only unnecessary components (ghosts) but also parallax components remain (extracted). Therefore, it is necessary to separate the parallax component remaining in the images of FIGS. 9 (q) to 9 (t) and the unnecessary component (ghost). Next, a method for separating a parallax component and an unnecessary component (ghost) will be described with reference to FIGS. FIG. 10 is an explanatory diagram of a method for separating a parallax component and an unnecessary component.

  10 (a), (b), (c), and (d) (the same images as those in FIGS. 9 (q), (r), (s), and (t)) are the above-described unnecessary components (ghost). And a parallax component. 10A to 10D are respectively compared to extract a common luminance component (common luminance portion). Here, as described above, only a portion having the same luminance value may be extracted as the common luminance portion. Alternatively, a predetermined luminance difference (threshold) may be provided as a parameter so that a certain amount of error is allowed and a luminance difference of ± X% when comparing luminance values is determined to be a common luminance portion. Good. For example, FIG. 10E shows a result (image COM1) of comparing the images MAX1 and MAX2 in FIGS. 10A and 10B and extracting a common luminance portion of these images. FIG. 10F shows a result (image COM2) obtained by comparing the images MAX3 and MAX4 in FIGS. 10C and 10D and extracting a common luminance portion of these images. FIG. 10G shows the result (image COM3) of comparing the images MAX2 and MAX4 in FIGS. 10B and 10D and extracting the common luminance portion of these images. FIG. 10H shows a result (image COM4) obtained by comparing the images MAX1 and MAX3 in FIGS. 10A and 10C and extracting a common luminance portion.

  FIG. 10I is an image COMtotal generated by adding the images COM1 to COM4 (direction component images) of FIGS. 10E to 10H. In the image COMtotal of FIG. 10 (i), the unnecessary component (ghost) and the parallax component are already separated. Moreover, when the number of pupil divisions is relatively small, such as two in the vertical direction and two in the horizontal direction as in the present embodiment, the oblique parallax component may not be completely removed. In such a case, it is preferable to add a process for expanding the region of the parallax component extracted from the image COMtotal of FIG.

  FIG. 11A shows an image COMtotal2 obtained by expanding a region including a parallax component in the image COMtotal of FIG. The object of the present embodiment is to separate the parallax component and the unnecessary component (ghost) from the mixed image (images MAX1 to MAX4) of the parallax component and the unnecessary component (ghost). For this reason, it suffices if either one of the parallax component and the unnecessary component can be erased by mask processing described later. Therefore, it is not always necessary to strictly extract the parallax component area, and a certain degree of effect can be obtained only by specifying a rough area. In addition, when the region including the parallax component is expanded, the mask may be modified so as to change the degree of expansion according to the subject distance using the distance information to the subject. Thereby, since the expansion amount can be changed according to the size of the blur, it becomes possible to separate the parallax components more precisely, and a great effect can be obtained.

  FIG. 11B shows a composite image (images obtained by averaging the images MAX1 to MAX4) of the images MAX1 to MAX4 (images in which the parallax component and the unnecessary component (ghost) are mixed) in FIGS. 10A to 10D. This is an image obtained by subtracting the image COMtotal2 of FIG. That is, the image of FIG. 11B is obtained from the synthesized image (averaged image) of the images MAX1 to MAX2 of FIGS. 10A to 10D using the image of FIG. 11A as a mask (mask image). This is an image MAXtotal-COMtotal2 generated by subtracting only the parallax component. In FIG. 11 (b), the luminance value of the unnecessary component (ghost) is drawn to be the same as that in FIG. 10 (c) for easy understanding. Since the average value of (d) is calculated, it becomes darker by that amount. Through the above procedure, the parallax component and the unnecessary component (ghost) can be separated from the parallax images (FIGS. 9A to 9D).

  Then, correction processing for removing or reducing unnecessary components (ghosts) determined as described above is performed on the image to be output. Thereby, as shown in FIG. 12, it is possible to obtain parallax images in which unnecessary components (ghosts) are substantially eliminated (removed or reduced). FIGS. 12A to 12D show parallax images Im10 to Im40 from which unnecessary components are removed or reduced, respectively, of FIGS. 9A to 9D. Further, by synthesizing the respective parallax images in which unnecessary components (ghosts) are reduced, as shown in FIG. 13, a captured image generated by imaging without pupil division with reduced unnecessary components (ghosts) and An equivalent image can be generated.

  Next, with reference to FIG. 14, the procedure of the determination process (image process) of the unnecessary component (ghost component) in a present Example is demonstrated. FIG. 14 is a flowchart illustrating an image processing method (unnecessary component determination method) in the present embodiment. Each step of FIG. 14 is mainly executed by the system controller 210 or the image processing unit 204 shown in FIG. 4 according to an image processing program as a computer program.

  First, in step S <b> 11, the system controller 210 controls an imaging unit including the imaging optical system 201 and the imaging element 202 to image a subject (acquires an input image as a captured image). In step S12, the system controller 210 controls the image processing unit 204, and uses the digital signal output from the image sensor 202 (pixel groups G1 to G4) and A / D converted by the A / D converter 203. Then, a pair of parallax images are generated as input images. Here, the image processing unit 204 may perform normal development processing and various image correction processing in order to generate a parallax image.

  Subsequently, in step S13, the first unnecessary component detection unit 204a of the image processing unit 204 obtains relative difference information (relative difference image) of each parallax image. That is, the first unnecessary component detection unit 204a uses the parallax images Im1 to Im4 in FIGS. 9A to 9D as reference images, and the relative difference images Im1-2 to Im4 in FIGS. -3 is generated. When unnecessary light that has reached the imaging surface passes through different pupil regions of the pupil of the imaging optical system 201, as shown in FIGS. 9A to 9D, an unnecessary component (ghost) is generated for each parallax image. The occurrence and location of occurrence are different. Therefore, in the simple relative difference image, the difference value of the unnecessary component (ghost) takes a positive value and a negative value. For example, in this embodiment, when the parallax image Im1 in FIG. 9A is subtracted from the parallax image Im3 in FIG. 9C to generate the relative difference image Im3-1 in FIG. The unnecessary components included in the parallax image Im3 of FIG. 9C is subtracted from the parallax image Im4 in FIG. 9D to generate the relative difference image Im4-3 in FIG. 9P, the parallax image in FIG. 9C is obtained. An unnecessary component included in Im3 has a negative value.

  In the present embodiment, the first unnecessary component detection unit 204a performs a process of rounding down the negative value to a zero value in order to simplify the unnecessary component reduction process described later. Therefore, in the relative difference images Im3-1, Im3-2, and Im3-4 in FIGS. 9G, 9K, and 9O, unnecessary components included in the parallax image Im3 in FIG. 9C are positive. Detected as a value. By performing similar processing for other relative difference images, only unnecessary components (ghosts) and parallax components included in the parallax images Im1 to Im4 in FIGS. 9A to 9D are detected as positive values. Will be.

  Subsequently, in step S14, the first unnecessary component detection unit 204a determines the component remaining in the relative difference image generated in step S13 as the first unnecessary component. Specifically, the first unnecessary component detection unit 204a obtains the relative difference image Im1-2 in FIGS. 9 (e), (i), and (m) obtained using the parallax image Im1 in FIG. 9 (a) as a reference image. , Im1-3 and Im1-4 are respectively compared. The unnecessary component detection unit 204a determines the first unnecessary component (image MAX1) by extracting the maximum luminance value on the same coordinates of each image. Further, when the first unnecessary component detection unit 204a performs the same processing on the relative difference images obtained by using the parallax images Im2, Im3, and Im4 in FIGS. 9B to 9D as reference images, FIG. Images MAX2 to MAX4 (first unnecessary components) of r) to (t) are obtained.

  Subsequently, in step S15, the direction component generation unit 204b of the image processing unit 204 compares the first unnecessary components (images MAX1 to MAX4) determined in step S14. Then, the direction component generation unit 204b extracts a common luminance component on the same coordinates of each first unnecessary component (each image) and extracts direction component images (for example, images COM1 to COM4 in FIGS. 10E to 10H). Is generated. At this time, only components having the same luminance value may be extracted, or if a luminance difference is ± X% when luminance values are compared while allowing a certain amount of error, a common luminance portion (common luminance component) A predetermined threshold value may be provided as a parameter so that it is determined that. At this time, in this embodiment, it is preferable to set X so that the common luminance component is determined when the luminance difference is ± X% (0 ≦ X ≦ 50). When the allowable amount of error is made smaller, X may be set so that, for example, 0 ≦ X ≦ 30.

  In step S16, the mask creation unit 204c of the image processing unit 204 generates a mask (mask image) based on the direction component image (direction component information) generated in step S15. At this time, the mask creation unit 204c can use the direction component image itself as a mask (for example, the image COMtotal in FIG. 10 (i)), or correct the mask by performing processing such as region enlargement. Good (for example, image COMtotal2 in FIG. 11A).

  Subsequently, in step S17, the second unnecessary component detection unit 204d of the image processing unit 204 determines a second unnecessary component based on the first unnecessary component and the mask. For example, the second unnecessary component detection unit 204d subtracts the mask (the image COMtotal2 in FIG. 11A) from the first unnecessary component (the average image of the images MAX1 to MAX4 in FIGS. 10A to 10D). . As a result, a second unnecessary component (image MAXtotal-COMtotal2 in FIG. 11B) is generated. Thereby, only the parallax component can be removed or reduced from the image in which the parallax component and the unnecessary component (ghost) are mixed.

  Subsequently, in step S18, the unnecessary component reduction unit 204e of the image processing unit 204 performs a correction process for reducing or removing unnecessary components from the image to be output. As shown in FIGS. 12A to 12D, the unnecessary component reduction unit 204e outputs a plurality of parallax images Im10 to Im40 obtained by setting each of the pixels G1 to G4 as one pixel. Is generated. Here, by discarding negative values to zero values in step S13, only unnecessary components (ghosts) included in each parallax image are detected as positive values. Therefore, the unnecessary component reduction unit 204e can remove or reduce the unnecessary component (ghost component) by simply subtracting the second unnecessary component from each parallax image. The unnecessary component reducing unit 204e subtracts the second unnecessary components (images MAX10 to MAX40) in FIGS. 15E to 15H from the parallax images Im1 to Im4 in FIGS. 15A to 15D, for example. Thereby, the unnecessary component reducing unit 204e can generate the parallax images Im10 to Im40 of FIGS. 15 (i) to (l) from which unnecessary components (ghost components) are removed or reduced.

  Finally, in step S19, the system controller 210 outputs the parallax images Im10 to Im40 from which unnecessary components (ghost components) are removed or reduced as shown in FIGS. 15 (i) to 15 (l) as output images. Then, the system controller 210 stores these output images in the image recording medium 209 or displays them on the display unit 205. Further, the parallax images Im10 to Im40 from which unnecessary components (ghost components) are removed or reduced are synthesized, and are equivalent to a captured image generated by imaging without pupil division from which unnecessary components are removed or reduced as shown in FIG. An image can also be output as an output image.

  According to the present embodiment, an unnecessary component (ghost component) formed by unnecessary light can be determined from a plurality of relative difference images based on a plurality of parallax images obtained by one imaging. That is, it is possible to determine an unnecessary component included in a captured image without performing multiple imaging. In addition, since negative values are rounded down when generating a relative difference image, a high-quality captured image in which unnecessary components determined only by simple difference calculation can be removed or reduced can be obtained.

  Next, a second embodiment of the present invention will be described. The basic configuration of the image pickup apparatus in the present embodiment is the same as that of the image pickup apparatus 200 of Embodiment 1 described with reference to FIG.

  FIG. 16 shows a light receiving portion of the image sensor 202a in the imaging system of the present embodiment. In FIG. 16, ML is a microlens, G11 to G55 are light receiving portions (pixels G11 to G55), and form a pair with each other. A plurality of pixel groups of pixels G11 to G55 are arranged on the image sensor 202a. This pixel group has a conjugate relationship with the exit pupil EXP via a common microlens ML (that is, one pixel group is provided for each pixel group). In this embodiment, a specific configuration example of the imaging optical system is also the same as that of the imaging optical system 201 of Embodiment 1 described with reference to FIG.

  FIG. 17 shows a region through which the light beam incident on the pixels G11 to G55 in FIG. 16 passes through the stop STP (that is, the exit pupil EXP of the imaging optical system, but the stop STP and the exit pupil EXP are actually different in many cases). P11 to P55 (pupil regions) are shown. The light beam from the high-intensity object passes through almost the entire area of the stop STP, but the region through which the light beam incident on each pixel passes is divided into pupil regions P11 to P55.

  Next, a method for determining an unnecessary component (ghost component) that appears due to photoelectric conversion of unnecessary light in a captured image generated by the imaging apparatus will be described with reference to FIGS. 18 and 19. FIG. 18 is a diagram illustrating a light receiving portion of the image sensor in the present embodiment. FIG. 19 is a diagram illustrating a procedure of an image processing method in the present embodiment.

  FIG. 19A shows a captured image Im51 generated by imaging without pupil division, which is the same as the captured image Im0 in FIG. The description of the same part as in the first embodiment also applies to a series of procedures such as obtaining a relative difference image using a set of parallax images obtained as a result of photoelectric conversion in the pixel groups G11 to G55. Omitted. The present embodiment is different from the first embodiment in that the oblique parallax component can be separated by increasing the number of oblique pupil divisions. For example, let us consider a case where an unnecessary component (ghost) ideally passes only a part of the pupil division region and enters the pixel G42 in FIG. 18 (the black square portion is a ghost).

  FIGS. 19B and 19A respectively show the parallax image Im42 corresponding to the pixel G42 and the parallax image Im51 corresponding to the outer pixel G51. Then, through steps S11 to S16 as shown in the flowchart of FIG. 14, a mask like the image COMtotal10 of FIG. 19C can be created. When the second unnecessary component is determined based on this mask and the unnecessary component (ghost) is reduced from the parallax image, an output image Im00 in which the unnecessary component is reduced as shown in FIG. 19D is obtained.

  According to the present embodiment, it is possible to separate parallax components in an oblique direction (a third direction different from the first direction (vertical direction) and the second direction (horizontal direction)). However, even if the number of pupil divisions is increased, information on the most peripheral part cannot be separated. This is because the parallax information that can be seen only from the most peripheral pupil position is included, and there is a common luminance component compared to the first unnecessary component image obtained with any pupil division position reference Because it does not.

  Therefore, in order to avoid this, it is preferable to set the pixel area for determining the second unnecessary component wider than the pixel area for generating the output image. For example, as illustrated in FIG. 18, a pixel (pixel area for generating an output image) used for shooting is a central 3 × 3 area. On the other hand, in order to reduce the unnecessary component (ghost) in the most peripheral portion, a peripheral pixel portion is provided further than pixels used for photographing. For example, a pixel region for reducing unnecessary components (ghost) (a pixel region for determining the first unnecessary component or the second unnecessary component) is a 5 × 5 region.

  Next, Embodiment 3 of the present invention will be described. Ren. "Plenoptic Camera" has been proposed in "Light Field Photographic with a Hand-Held Plenoptic Camera" (Stanford Tech Report CTSR 2005-2) of Ng et al. By using the method “Light Field Photography” in “Plenoptic Camera”, it is possible to capture information on the position and angle of the light beam from the object side.

  FIG. 20 is a diagram illustrating an imaging system of the imaging apparatus according to the present embodiment, and illustrates a configuration of an imaging system of “Plenoptic Camera”. The imaging optical system 301 includes a main lens (photographing lens) 301b and an aperture stop 301a. A microlens array 301c is arranged at the imaging position of the imaging optical system 301, and an imaging element 302 is arranged behind (on the image side) of the microlens array 301c. The microlens array 301c serves as a separator (separating means) so that a ray group passing through a certain point in the subject space such as the point A and a ray passing through a point near the point A are not mixed on the image sensor 302. It has a function. As can be seen from FIG. 20, the upper line, chief ray and underline from point A are received by different pixels. For this reason, a group of rays passing through the point A can be obtained separately for each angle of rays.

  Also, “Full Resolution Light Field Rendering” (Adobe Technical Report January 2008) by Todor George et al. Is known. This document proposes an imaging system shown in FIGS. 21 and 22 as a method of acquiring light position and angle information (Light Field).

  In the configuration of the imaging system shown in FIG. 21, the microlens array 301c is arranged behind (image side) the imaging position of the main lens 301b, and a light ray group passing through the point A is re-imaged on the imaging device 302. Thus, the light beam group can be obtained separately for each light beam angle. In the configuration of the imaging system shown in FIG. 22, the microlens array 301c is arranged in front of the imaging position of the main lens 301b (object side), and a group of rays passing through the point A is imaged on the imaging element 302. By doing so, the light beam group can be obtained separately for each angle of the light beam. Both configurations are the same in that the light beam passing through the pupil of the imaging optical system 301 is divided according to the passing region (passing position) in the pupil. In these configurations, the imaging element 302 uses a conventional imaging element that is paired via one microlens ML, one light receiving unit G1, and a color filter CF, as shown in FIG. Can do.

  When the imaging optical system 301 shown in FIG. 20 is used, an image as shown in FIG. 24A is obtained. FIG. 24B shows an enlarged view of one of many circles arranged in FIG. One circle corresponds to the stop STP, and the inside thereof is divided by a plurality of pixels Pj (j = 1, 2, 3,...). Thereby, the intensity distribution of the pupil is obtained within one circle. Further, when the imaging optical system 301 shown in FIGS. 21 and 22 is used, a parallax image as shown in FIG. 25 is obtained. In the image shown in FIG. 24A, a plurality of parallax images as shown in FIG. 25 can also be obtained by rearranging a plurality of pixels Pj in each circle (aperture STP) side by side.

  As described in the first and second embodiments, unnecessary light such as ghost passes through the pupil with a bias in the pupil. For this reason, an unnecessary component can also be determined by using the image processing method described in the first and second embodiments in an imaging apparatus that divides and images the pupil as in the present embodiment.

  As another example, a parallax image can be obtained even when the same subject is imaged using a plurality of cameras as shown in FIG. For this reason, the image processing method described in the first and second embodiments can be used in such a plurality of cameras. C1, C2, and C3 are actually separate imaging devices, but can be regarded as an integrated imaging device that divides a large pupil into three images. In addition, as shown in FIG. 27, pupil division can be performed by providing a plurality of imaging optical systems OSj (j = 1, 2, 3,...) In one imaging apparatus.

  Next, a fourth embodiment of the present invention will be described. In each of the above-described embodiments, the case where the unnecessary component (ghost component) is determined or removed in the entire image has been described. However, in many cases, the unnecessary component is only a part of the image as shown in FIG. Arise. Since it is easy for the user to determine the unnecessary component area in the image, the processing load in each embodiment can be reduced by causing the user to specify an image area to be reduced. Further, by limiting the area, it is possible to reduce the influence of the subject parallax component that occurs in the case of the above-mentioned short-distance subject shooting.

  FIG. 28 is an explanatory diagram of a region where unnecessary component reduction processing is performed. FIG. 28A shows an image when the user selects a region (image region) from which an unnecessary component (ghost) is to be removed in the image. FIG. 28B shows an output image obtained by reducing unnecessary components (ghosts) in the solid line image area (selected area). Thus, even when the image area from which the unnecessary component is removed is limited to a part of the image area, the unnecessary component (ghost) is determined or removed from the parallax image using the same image processing method as in the first and second embodiments. be able to.

  In each embodiment, the case where the second unnecessary component (ghost component) is removed or reduced has been described. However, by using information on the determined unnecessary component (ghost component), another unnecessary component (third unnecessary component) is used. Correction processing for adding (or emphasizing) may be performed. For example, for each of the plurality of parallax images shown in FIG. 25, an image in which an unnecessary component (ghost component) is present and an image in which it is not present are generated. When an unnecessary component (ghost component) is desired to remain in the reconstructed image, the determined unnecessary component (ghost component) may be added to each parallax image. In addition, an unnecessary component (ghost component) can be added to the reconstructed image.

  In each of the embodiments, the imaging apparatus using the image processing method of the present invention (mounted with the image processing apparatus) has been described. However, the image processing method of the present invention is also implemented by an image processing program installed in a personal computer. be able to. In this case, the personal computer corresponds to the image processing apparatus of the present invention. The personal computer captures (acquires) an image (input image) before image recovery processing generated by the imaging device, performs image recovery processing by an image processing program, and outputs the resulting image.

  As described above, in each embodiment, the image processing apparatus (image processing unit 204) includes first unnecessary component determination means (first unnecessary component detection unit 204a) and direction component determination means (direction component generation unit 204b). The image processing apparatus also includes a mask creation unit (mask creation unit 204c) and a second unnecessary component determination unit (second unnecessary component detection unit 204d). The first unnecessary component determining means determines a plurality of first unnecessary components related to each of the plurality of parallax images based on the relative difference information of the plurality of parallax images (step S14). The direction component determining means determines direction component information (direction component image) based on the plurality of first unnecessary components (step S15). The mask creating means creates a mask (mask image) based on the direction component information (step S16). The second unnecessary component determining means determines a second unnecessary component based on the plurality of first unnecessary components and the mask (step S17).

  Preferably, the direction component information is determined by comparing luminance components at the same coordinate position among the plurality of first unnecessary components. More preferably, the direction component information is determined based on whether or not the luminance component at the same coordinate position is a common luminance component. Preferably, when the luminance difference between the luminance components at the same coordinate position is within a predetermined luminance difference range, the luminance component is determined to be a common luminance component. More preferably, the predetermined luminance difference is ± X% (0 ≦ X ≦ 50).

  Preferably, the image processing unit generates a plurality of parallax images by one imaging operation (step S12). Preferably, the image processing unit sets each of the plurality of parallax images as a reference image, calculates a difference between the other parallax image excluding the reference image and the reference image among the plurality of parallax images, and outputs a plurality of relative differences. Information is obtained (step S13).

  Preferably, the first unnecessary component determination unit truncates a negative value of the plurality of pieces of relative difference information obtained using, as a reference image, a parallax image that is a target for determining the first unnecessary component among the plurality of parallax images. Obtained as two-dimensional data. Then, the first unnecessary component determining means extracts a maximum value between a plurality of pieces of relative difference information at each position of the two-dimensional data, and determines the position and amount of the first unnecessary component in the reference image.

  Preferably, the image processing unit includes unnecessary component reducing means (unnecessary component reducing unit 204e). The unnecessary component reducing means generates an output image by subtracting the second unnecessary component corresponding to the plurality of parallax images from the plurality of parallax images (step S19). More preferably, the pixel area for determining the second unnecessary component is wider than the pixel area for generating the output image.

  Preferably, the image processing unit selects a determination target region for the second unnecessary component. And a 2nd unnecessary component determination means determines a 2nd unnecessary component in the selected determination object area | region. Preferably, the plurality of parallax images are a plurality of images generated based on a plurality of light beams that have passed through different regions of the pupil of the imaging optical system. Preferably, the image processing unit performs correction processing for adding the third unnecessary component to at least one of the plurality of parallax images using the information regarding the second unnecessary component. Preferably, the mask creating means corrects the mask according to the subject distance.

  Preferably, the direction component information includes parallax information (or information corresponding to the parallax information) in a first direction (vertical direction) and a second direction (lateral direction) different from the first direction. More preferably, the direction component information further includes parallax information (or information corresponding to the parallax information) in a third direction (oblique direction) different from both the first direction and the second direction.

  According to each embodiment, an image processing method, an image processing device, an imaging device, and an image processing capable of separating a parallax component and an unnecessary component included in a captured image without performing a plurality of images (with one imaging operation) A program and a storage medium can be provided.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

204 Image processing unit (image processing apparatus)
204a First unnecessary component detection unit (first unnecessary component determination means)
204b Direction component generation unit (direction component determination means)
204c Mask creation part (mask creation means)
204d 2nd unnecessary component detection part (2nd unnecessary component determination means)

Claims (20)

Determining a plurality of first unnecessary components for each of the plurality of parallax images based on a plurality of pieces of relative difference information of the plurality of parallax images;
Determining direction component information based on the plurality of first unnecessary components;
Image processing method characterized by comprising the steps of: determining a second required component on the basis of the previous SL plurality of first unnecessary component and the direction component information. The image processing according to claim 1, wherein the first unnecessary component related to each of the plurality of parallax images includes a component related to parallax and a component generated when unnecessary light reaches the imaging surface. Method. The relative difference information of the plurality of parallax images sets each of the plurality of parallax images as a reference image, and the difference between each of the other parallax images excluding the reference image among the plurality of parallax images and the reference image The image processing method according to claim 1, wherein the image processing method is information obtained by calculating. The image processing method according to claim 1, wherein the direction component information is parallax information between at least two parallax images in the plurality of parallax images. The image processing method according to claim 1, wherein the direction component information includes parallax information in two different directions. The image processing method according to claim 1, wherein the direction component information further includes parallax information in three different directions. The image processing method according to claim 1, wherein the second unnecessary component is a component generated when unnecessary light reaches the imaging surface. The directional component information, the image processing method according to any one of claims 1 to 7, characterized in that it is determined by comparing the luminance components at the same coordinate positions in the plurality of first unnecessary components. The image processing method according to claim 8 , wherein the direction component information is determined based on whether or not the luminance component at the same coordinate position is a common luminance component. The image processing according to claim 9 , wherein when the difference between the luminance components at the same coordinate position is within a predetermined luminance difference range, the luminance component is determined to be the common luminance component. Method. In the step of determining the plurality of first unnecessary components,
Acquiring the plurality of relative difference information obtained parallax image as a target for determining said plurality of parallax images sac Chi before Symbol first required component as a reference image as a two-dimensional data the truncated negative value,
The extracted maximum value between Oite each relative difference information at each position of the two-dimensional data to determine the first required Ingredients in the reference image, any one of claims 1 to 10, characterized in that 2. The image processing method according to item 1. According to any one of claims 1 to 11, further comprising the step of generating an output image by subtracting the second required component corresponding to the plurality of parallax images from the plurality of parallax images Image processing method . The picture element region that determine the second unnecessary component, image processing method according to claim 12, wherein the wider than the pixel area to produce the output image. In determining a pre Symbol second required component, at least one or said second unnecessary component for a partial region in the obtained synthesized image by synthesizing the plurality of parallax images of the plurality of parallax images the image processing method according to any one of claims 1 to 13, characterized in that determining. Wherein the plurality of parallax images into a plurality of any one of claims 1 to 14, characterized in that a plurality of images generated based on the light beams passing through different areas of the pupil of the imaging optical system The image processing method as described. 16. The method according to claim 1, further comprising a step of performing a correction process of adding a ghost component to at least one of the plurality of parallax images using the information related to the second unnecessary component. The image processing method as described. First unnecessary component determining means for determining a plurality of first unnecessary components for each of the plurality of parallax images based on a plurality of pieces of relative difference information of the plurality of parallax images;
Direction component determining means for determining direction component information based on the plurality of first unnecessary components;
The image processing apparatus characterized by having, a second unnecessary component determining means for determining a second required component on the basis of the previous SL plurality of first unnecessary component and the direction component information. An image pickup apparatus comprising : an image pickup element that photoelectrically converts an optical image and outputs a plurality of parallax images; and the image processing apparatus according to claim 17 . Determining a plurality of first unnecessary components related to each of the plurality of parallax images based on a plurality of pieces of relative difference information of the plurality of parallax images;
Determining direction component information based on the plurality of first unnecessary components;
The image processing program characterized by being configured to execute determining a second required component on the basis of the previous SL plurality of first unnecessary component and the direction component information, to the computer.   A storage medium storing the image processing program according to claim 19.