JP5083574B2 - Image processing apparatus and program - Google Patents

Description

  The present invention relates to an image processing apparatus and a program for detecting foreign matter in an image.

  In an imaging apparatus such as a digital camera, foreign matter may adhere to an optical member such as an optical filter disposed on the imaging optical path, and an image defect may occur in a captured image due to the attached foreign matter. As a conventional technique for solving such a problem, for example, there has been proposed an image processing apparatus that detects a foreign substance reflected in an image based on luminance data of a captured image and removes the foreign substance (Patent Document). 1 etc.).

JP 2004-220553 A

  However, in the prior art, the detection accuracy of a foreign object reflected in a captured image may be insufficient, and erroneous detection may occur. Further, the conventional technique has a problem that it is difficult to determine whether or not the luminance change in the specific area is due to the reflection of a foreign object.

  The present invention has been made in view of such a situation, and an object thereof is to provide an image processing apparatus and an image processing program for accurately detecting a foreign object in an image.

In order to achieve the above object, an image processing apparatus according to the present invention provides:
An image input unit (28) for inputting a captured image;
Using the brightness information corresponding to the brightness of the detection area in the captured image and the saturation information corresponding to the saturation of the detection area, the detection area corresponds to a foreign object reflected in the captured image. A determination unit (48) for determining whether or not it is a part, and after the determination unit determines whether or not the detection region is a part corresponding to the foreign object using the brightness information, Using the saturation information, it is determined whether or not the detection area is a portion corresponding to the foreign matter.

  In addition, for example, the determination unit determines in advance a difference between the first saturation information corresponding to the saturation of the detection region and the second saturation information corresponding to the saturation of the region around the detection region. If it is within the range, the detection area may be determined to be a portion corresponding to the foreign matter.

  In addition, for example, the determination unit determines in advance a difference between the first saturation information corresponding to the saturation of the detection region and the second saturation information corresponding to the saturation of the region around the detection region. If it is not within the range, it may be determined that the detection area is not a part corresponding to the foreign matter.

  In addition, for example, the determination unit determines whether the detection region is a portion corresponding to the foreign object by using a gradient between the brightness of the detection region and the brightness of a region around the detection region. Also good.

  In addition, for example, the determination unit uses the ratio between the first saturation information corresponding to the saturation of the detection region and the second saturation information corresponding to the saturation of the region around the detection region. It may be determined whether or not the detection region is a portion corresponding to the foreign matter.

  Further, for example, the determination unit determines that the detection area is not a part corresponding to the foreign object when the second brightness information corresponding to the brightness of the area around the detection area is not a predetermined range. May be.

  Further, for example, the determination unit may determine that the detection area is not a part corresponding to the foreign object when a standard deviation of brightness of the area around the detection area is not in a predetermined range.

The program according to the present invention is:
On the computer,
The procedure to input the shot image,
A procedure for calculating brightness information of the captured image;
A procedure for calculating saturation information of the captured image;
And a procedure for detecting foreign matter information related to the foreign matter reflected in the image using the brightness information and the saturation information.

A program according to the second aspect of the present invention is:
On the computer,
The procedure to input the shot image,
Using the brightness information corresponding to the brightness of the detection area in the captured image and the saturation information corresponding to the saturation of the detection area, the detection area corresponds to a foreign object reflected in the captured image. And a procedure for determining whether or not it is a part.

In the above description, in order to explain the present invention in an easy-to-understand manner, the description is made in association with the reference numerals of the drawings showing the embodiments. However, the present invention is not limited to this. The configuration of the embodiment described later may be improved as appropriate, or at least a part of the configuration may be replaced with another component. Further, the configuration requirements that are not particularly limited with respect to the arrangement are not limited to the arrangement disclosed in the embodiment, and can be arranged at a position where the function can be achieved.

FIG. 1 is a schematic sectional view of a camera including an image processing apparatus according to an embodiment of the present invention. FIG. 2 is an overall block diagram of the camera shown in FIG. FIG. 3A is a flowchart illustrating the first half of an example of the image processing calculation executed by the image processing apparatus according to the present embodiment. FIG. 3B is a flowchart illustrating the latter half of an example of the image processing calculation executed by the image processing apparatus according to the present embodiment. FIG. 4 is a conceptual diagram showing the process of creating a gain map. FIG. 5 is a conceptual diagram showing a foreign matter detection level adjustment process in the image processing apparatus according to the present embodiment. FIG. 6 is a conceptual diagram showing images and the like calculated in the image processing apparatus according to the present embodiment. FIG. 7 is a block diagram of a computer including an image processing apparatus according to the second embodiment of the present embodiment.

First Embodiment FIG. 1 is a schematic view of a camera 10 including an image processing apparatus according to an embodiment of the present invention. The camera 10 includes a lens barrel portion 14 provided with an optical system 16 and the like, and a camera body portion 12 provided with an imaging element 22 and the like. In the present embodiment, an example of an apparatus including an image processing apparatus will be described, which is an interchangeable lens camera in which the lens barrel section 14 is detachably attached to the camera body section 12, but the image processing apparatus according to the present embodiment is provided. The equipment to be used is not limited to this. For example, a camera in which the lens barrel unit 14 and the camera body unit 12 are integrated, a video camera, a mobile phone, a PC (personal computer) that can process image data, or the like may be used.

  The camera body 12 shown in FIG. 1 includes a shutter 18, an optical filter 20, an image sensor 22 and the like. The shutter 18 adjusts the exposure time by shielding and passing photographing light from the optical system 16 toward the image sensor 22. The optical filter 20 is configured by an optical low-pass filter or the like that prevents the occurrence of false colors (color moire) during imaging.

  The image sensor 22 can acquire image data by photoelectrically converting the light guided by the optical system 16. However, as shown in FIG. 1, when foreign matter 24 such as dust adheres to a member such as the optical filter 20 that is on the optical path of the imaging light and transmits the imaging light, the image data acquired by the imaging element 22 In some cases, pixels affected by the foreign matter 24 such as dust are included. The pixels affected by the foreign matter 24 include, for example, pixels whose color (lightness, etc.) has changed from the original color of the photographing light as a result of the passage of photographing light being blocked by the foreign matter 24. As will be described later, the camera 10 according to the present embodiment can perform image processing for detecting pixels affected by the foreign matter 24. In addition, as described later, the camera 10 can perform image processing for reducing or removing the influence of the foreign matter 24 on the captured image including the pixels affected by the foreign matter 24.

  FIG. 2 is a schematic block diagram showing the configuration of the image processing apparatus provided in the camera 10 shown in FIG. The camera 10 includes a photographing unit 26 including the image sensor 22 shown in FIG. 1 and an image processing device that performs image processing on a photographed image photographed by the photographing unit 26 and the like. The image processing apparatus mounted on the camera 10 includes an image conversion unit 115, an image input unit 28, a storage unit 30, an operation signal input unit 32, an imaging condition signal input unit 34, a display unit 36, a calculation unit 38, and the like.

  The image input unit 28 receives the captured image captured by the imaging unit 26 in the form of image data. The captured image is RGB data, for example, but the data format of the captured image is not particularly limited. The image input unit 28 outputs the input captured image to the calculation unit 38. The computing unit 38 performs image processing on the captured image input via the image input unit 28. Note that the image input unit 28 may receive not only a photographed image photographed by the photographing unit 26 but also a photographed image photographed by another photographing apparatus. In addition, the captured image on which the calculation unit 38 performs image processing is not limited to the captured image captured by the capturing unit 26, and includes captured images captured by other imaging devices.

  Further, the image input unit 28 includes not only a photographed image obtained by photoelectrically converting an image obtained by the optical system 16 but also a photographed image that has been subjected to image processing in which the photographed image is subjected to γ correction by the image conversion unit 115. Entered. For example, the image conversion unit 115 performs image processing such as γ correction, change of a storage format, data compression processing, and the like on a captured image obtained by photoelectrically converting an image obtained by the optical system 16. Accordingly, the image conversion unit 115 can correct the captured image so that the captured image is displayed more vividly or in a more natural color when the captured image is displayed on a display device such as a liquid crystal display. . Further, as the image conversion unit 115 compresses the captured image, the camera 10 can store more captured images in a memory card (not shown) or the like. Image processing performed on the captured image by the captured image conversion unit 115 includes linear correction and nonlinear correction.

  The calculation unit 38 includes an information calculation unit 46, a foreign matter determination unit 48, and a foreign matter image correction unit 50. The information calculation unit 46 can analyze the captured image and calculate data regarding the color of the captured image. Using the data calculated by the information calculation unit 46, the foreign matter determination unit 48 can determine whether or not the predetermined detection area is a portion corresponding to foreign matter such as dust reflected in the captured image. . Further, the foreign object image correcting unit 50 identifies a part corresponding to the foreign object reflected in the captured image based on the determination by the foreign object determining unit 48, and performs image processing for reducing the influence of the foreign object on the captured image. be able to. In addition, the foreign object image correction unit 50 can perform image processing that reduces the influence of the foreign object, using the data calculated by the information calculation unit 46.

  The information calculation unit 46 includes a brightness information calculation unit 54, a saturation information calculation unit 56, a hue information calculation unit 58, a brightness gradient information calculation unit 60, and a region calculation unit 62. The calculation units 54, 56, 58, 60 and 62 included in the information calculation unit 46 perform arithmetic processing on the captured image, and calculate different data for each of the calculation units 54, 56, 58, 60 and 62.

  The brightness information calculation unit 54 calculates brightness information corresponding to the brightness of the captured image. The brightness information calculated by the brightness information calculation unit 54 may be brightness information of the captured image, brightness information of the captured image, or both brightness information and brightness information.

  The brightness information calculation unit 54 calculates the luminance value Y of each pixel from the RGB value of each pixel in the captured image using, for example, the following equation (1).

  Also, the brightness information calculation unit 54 generates a brightness plane corresponding to the captured image from the brightness value Y of the pixel. Therefore, the luminance information calculated by the brightness information calculation unit 54 includes the luminance value Y of each pixel, and the luminance plane constituted by the position and luminance value Y of each pixel.

  The brightness information calculation unit 54 may calculate the brightness information instead of the brightness information or together with the brightness information. The brightness information is obtained by HSV conversion of RGB data. HSV conversion converts RGB data into hue (Hue), saturation (Saturation), and lightness (Value). The brightness value V, hue value H, and saturation value S of each pixel in the captured image can be obtained from the RGB values of each pixel by the following formulas (2-1) to (2-10).

  Note that when H <0 as a result of conversion by the equations (2-3) to (2-5), 2π is added to H. When Vmax = 0, S = 0 and H = undefined.

  The brightness information calculation unit 54 calculates the brightness value V of each pixel from the RGB value of each pixel in the captured image using the above equations (2-1) and (2-6). Also, the brightness information calculation unit 54 generates a brightness plane corresponding to the captured image plane from the brightness value V of the pixel. Therefore, the brightness information calculated by the brightness information calculation unit 54 includes the brightness value V of each pixel, and the brightness plane constituted by the position and brightness value V of each pixel.

  The saturation information calculation unit 56 calculates saturation information corresponding to the saturation of the captured image. The saturation information calculation unit 56 uses, for example, the above equations (2-2), (2-6), and (2-7) to calculate the saturation value S of each pixel from the RGB value of each pixel in the captured image. Is calculated. Further, the saturation information calculation unit 56 generates a saturation plane corresponding to the captured image plane from the saturation value S of the pixel. Therefore, the saturation information calculated by the saturation information calculation unit 56 includes the saturation value S of each pixel, and the saturation plane constituted by the position and saturation value S of each pixel.

  The hue information calculation unit 58 calculates hue information corresponding to the hue of the captured image. The hue information calculation unit 58 uses, for example, the above equations (2-3) to (2-5) and (2-6) to (2-10) to calculate each pixel from the RGB value of each pixel in the captured image. Is calculated. Also, the hue information calculation unit 58 generates a hue plane corresponding to the captured image plane from the hue value H of the pixel. Therefore, the hue information calculated by the hue information calculation unit 58 includes the hue value H of each pixel and the hue plane formed by the position and hue value H of each pixel.

  The brightness gradient information calculation unit 60 calculates brightness gradient information corresponding to the brightness gradient of the captured image from the brightness plane and the brightness plane calculated by the brightness information calculation unit 54. The brightness gradient information calculated by the brightness gradient information calculation unit 60 includes brightness gradient information and brightness gradient information.

  In this case, it is preferable that the brightness gradient information calculation unit 60 calculates the brightness gradient information from the luminance plane and the brightness plane that have been subjected to inverse γ correction by the inverse γ correction unit 40. Since the captured image is often stored in a state of being γ-corrected according to the display device or the like, the captured image input to the calculation unit 38 via the image input unit 28 is also subjected to γ correction. There is a high possibility. Therefore, when the inverse γ correction unit 40 performs inverse γ correction on the luminance plane and the brightness plane, the brightness information can be brought close to the original brightness information that the RAW data has.

  If the luminance plane and lightness plane before reverse γ correction are YP1 and VP1, and the luminance plane and lightness plane after reverse γ correction are YP2 and VP2, the reverse γ correction performed by the reverse γ correction unit 40 is expressed by the following formula ( 3-1) and (3-2).

  The brightness gradient information calculation unit 60 calculates the brightness gradient value Vgrad (i, j) at each position (pixel) in the captured image, for example, from the brightness plane VP2. The brightness gradient information calculation unit 60 applies vertical and horizontal differential filters to the brightness plane VP2 to calculate a vertical difference Δyf (i, j) and a horizontal difference Δxf (i, j). Then, the brightness gradient value Vgrad (i, j) at each position is calculated using the calculated difference. When the lightness value V of each pixel included in the lightness plane VP2 is V (i, j), the lightness gradient value Vgrad (i, j) is calculated by the following equations (4-1) to (4-3). The

  Note that i in the expressions (4-1) to (4-3) represents the position in the horizontal direction of the pixel, and j represents the position in the vertical direction of the pixel. The brightness gradient information calculation unit 60 generates a brightness gradient plane corresponding to the captured image plane from the brightness gradient value Vgrad (i, j) at each position. Therefore, the lightness gradient information calculated by the lightness gradient information calculation unit 60 includes a lightness gradient value Vgrad (i, j) and a lightness gradient plane.

  Further, the brightness gradient information calculation unit 60 can calculate the brightness gradient information from the brightness plane YP2 instead of the brightness gradient information or in addition to the brightness gradient information. The luminance gradient information includes a luminance gradient value Ygrad (i, j) and a luminance gradient plane. The method of calculating the brightness gradient information is the same as the brightness gradient information described above.

  The area calculation unit 62 calculates a detection area for detecting whether or not a foreign object is reflected. The region calculation unit 62 can calculate the detection region using the brightness gradient information calculated by the brightness gradient information calculation unit 60, for example.

  The detection area is not particularly limited as long as it is a part of the photographed image, but an area where a foreign object is likely to be reflected is preferably set as the detection area. Here, in the lightness gradient plane calculated by the lightness gradient information calculation unit 60, the lightness gradient value Vgrad (i, j) tends to be relatively large in a portion corresponding to the subject edge or a portion in which a foreign object is reflected. It is in. On the other hand, the brightness gradient value Vgrad (i, j) tends to be relatively small in a portion corresponding to the sky or the inside of the subject.

  Therefore, the region calculation unit 62 according to the present embodiment calculates a binarized image obtained by binarizing the lightness gradient plane, and a connected region in which pixels having a high lightness gradient value Vgrad (i, j) are connected is obtained. It can be a detection region. That is, the region calculation unit 62 sets a predetermined threshold for the lightness gradient plane, replaces the value of a pixel having a lightness gradient value higher than the threshold with “1”, and has a lightness gradient value lower than the threshold By substituting the value of “0” with “0”, a binarized image is calculated.

  Furthermore, the area calculation unit 62 extracts pixels having a value “1” from the calculated binarized image, and sets the pixels having a value “1” as a detection area. If the pixels having the value “1” are connected to each other, the region calculation unit 62 specifies a group of these pixels as one detection region. On the other hand, the area calculation unit 62 requires that one pixel set having a value “1” is connected to another pixel set having a value “1” by pixels having a value “1”. For example, these two areas are specified as different detection areas.

  As described above, the area calculation unit 62 may calculate the detection area from the brightness gradient plane, but in addition to that, the area calculation unit 62 calculates the detection area from the luminance gradient plane calculated by the brightness information calculation unit 54. You can also Similar to the lightness gradient plane, the region calculation unit 62 can calculate the detection region by calculating a binarized image from the luminance gradient plane.

  Using the information calculated by the information calculation unit 46, the foreign object determination unit 48 determines whether each detection area is a part corresponding to a foreign object reflected in the captured image. The foreign matter determination unit 48 includes a first determination unit 64, a second determination unit 66, a third determination unit 68, and a fourth determination unit 70. The first to fourth determination units 64 to 70 determine whether or not the detection area to be determined is a portion corresponding to a foreign object reflected in the captured image using different determination criteria. .

  The first determination unit 64 determines whether or not the detection area is a part corresponding to a foreign object reflected in the image, based on the size of the detection area to be determined. In the image processing apparatus according to the present embodiment, the detection region to be determined by the foreign matter determination unit 48 and the size thereof are determined using the brightness gradient information calculated by the brightness gradient information calculation unit 60. 62. Therefore, for example, when the region calculation unit 62 calculates the detection region from the luminance gradient plane, the first determination unit 64 makes a determination using information such as the luminance gradient value of each pixel. When the area calculation unit 62 calculates the detection area from the lightness gradient plane, the first determination unit 64 makes a determination using information such as the lightness gradient value of each pixel.

  If the size of the detection region calculated by the region calculation unit 62 is within a predetermined range, the first determination unit 64 can determine that the detection region is a portion corresponding to a foreign object reflected in the image. it can. The first determination unit 64 reflects the detection region in the image if the size of the detection region calculated by the region calculation unit 62 is larger than the predetermined range or smaller than the predetermined range. It can be determined that the portion does not correspond to a foreign object.

  The range of the size determined by the first determination unit 64 to be a region where a foreign object is reflected is set according to the size of the captured image, the shooting conditions, etc. For example, the lower limit value is 25 pixels and the upper limit value is It can be 300 pixels.

  The second determination unit 66 uses the saturation information corresponding to the saturation of the detection area to determine whether or not the detection area is a part corresponding to a foreign object reflected in the image. The second determination unit 66 uses the saturation value S calculated by the saturation information calculation unit 56 and the saturation plane to determine whether or not the detection region is a part corresponding to a foreign object reflected in the image. ,to decide. The second determination unit 66 according to the present embodiment investigates a difference between the first saturation information corresponding to the saturation of the detection region and the second saturation information corresponding to the saturation of the region around the detection region. . When the difference between the first saturation information and the second saturation information is within a predetermined range, the second determination unit 66 determines that the detection area that is the determination target is a portion corresponding to a foreign object. To do. Further, when the difference between the first saturation information and the second saturation information is not in a predetermined range, the second determination unit 66 determines that the detection area to be determined is not a part corresponding to a foreign object. May be.

  Here, the first saturation information can be an average value of the saturation values S of the pixels included in the detection region, for example. Further, the second saturation information can be an average value of the saturation values S of the pixels in the area around the detection area. The region around the detection region can be defined as, for example, a region that surrounds the periphery of the detection region while being in contact with the outer periphery of the detection region, and has a similar area to the detection region. .

  The numerical range in which the second determination unit 66 determines that the detection area to be determined is an area in which a foreign object is reflected is not particularly limited. In addition, the numerical value range that is determined to be a region in which a foreign object is reflected may be defined by the magnitude (absolute value) of the difference between the saturation values S. The first saturation information and the second saturation information It may be defined by a ratio of values.

  For example, the second determination unit 66 determines that the average value S1 of the saturation values S of the pixels included in the detection area is 0.95 to 2 times the average value S2 of the saturation values S of the pixels in the surrounding area of the detection area. If it is, it is determined that the detection area to be determined is an area in which foreign matter is reflected. This is because an image reflected in an image as a subject instead of dust leaves strong information in the saturation information, and conversely, an image in which a foreign object is reflected has little effect on the saturation. Therefore, if dust does not appear in the detection area, the difference between the first saturation information and the second saturation information is large, and conversely, if dust appears in the detection area, the first saturation information. The difference between the second saturation information and the second saturation information is small.

  The second determination unit 66 not only uses the saturation information as described above, but also uses the brightness information corresponding to the brightness of the detection area to capture the detection area in the image. It may be determined whether or not the portion corresponds to a foreign object. When the determination is made using the brightness information, the second determination unit 66 uses the brightness value V calculated by the brightness information calculation unit 54 and the brightness plane to detect a foreign object whose detection area is reflected in the image. It can be determined whether or not it is a corresponding part. In addition, the second determination unit 66 uses the luminance value Y calculated by the brightness information calculation unit 54 and the luminance plane to determine whether or not the detection region is a part corresponding to a foreign object reflected in the image. You may judge.

  For example, the second determination unit 66 according to the present embodiment investigates a difference between the first lightness information corresponding to the lightness of the detection area and the second lightness information corresponding to the lightness of the area around the detection area. When the difference between the first lightness information and the second lightness information is not within a predetermined range, the second determination unit 66 determines that the detection area to be determined is a part corresponding to a foreign object.

  Here, the first brightness information can be, for example, an average value of the brightness values V of the pixels included in the detection region. Further, the second brightness information can be an average value of the brightness values V of the pixels in the area around the detection area. For example, the second determination unit 66 determines that the average value V1 of the brightness values V of the pixels included in the detection area is less than 0.975 times the average value V2 of the brightness values V of the pixels around the detection area. In addition, it is determined that the detection area to be determined is an area in which foreign matter is reflected. This is because the reflection of a foreign object has little effect on the saturation and hue, but leaves strong information on brightness.

  Further, in addition to the determination using the saturation information, the second determination unit 66 uses the hue information corresponding to the hue of the detection area to determine whether the detection area corresponds to a foreign object reflected in the image. It may be determined whether or not. When the determination is performed using the hue information, the second determination unit 66 uses the hue value H calculated by the hue information calculation unit 58 and the hue plane to correspond to a foreign object whose detection area is reflected in the image. It can be determined whether it is a part.

  For example, the second determination unit 66 according to the present embodiment investigates the difference between the first hue information corresponding to the hue of the detection area and the second hue information corresponding to the hue of the area around the detection area. When the difference between the first hue information and the second hue information is in a predetermined range, the second determination unit 66 determines that the detection area to be determined is a part corresponding to a foreign object.

  Here, the first hue information can be, for example, an average value of the hue values H of the pixels included in the detection region. Further, the second hue information can be an average value of the hue values H of the pixels in the area around the detection area. For example, the second determination unit 66 determines that the average value H1 of the hue values H of the pixels included in the detection area is 0.94 to 1.06 times the average value H2 of the hue values H of the pixels around the detection area. If it is, it is determined that the detection area to be determined is an area in which foreign matter is reflected. This is because an image reflected in an image as a subject instead of dust leaves strong information in the hue information, and conversely, an image in which a foreign object is reflected has little influence on the hue.

  When the second brightness information corresponding to the brightness of the surrounding area of the detection area is not in a predetermined range, the third determination unit 68 is not the part corresponding to the foreign object. Judge. In addition, when the second brightness information corresponding to the brightness of the area around the detection area is within a predetermined range, the third determination unit 68 is a part where the detection area to be determined corresponds to a foreign object. It is judged that. The brightness information used by the third determination unit 68 includes second brightness information, which is an average value of the brightness values V of the pixels around the detection area, and an average of the brightness values Y of the pixels around the detection area. Second luminance information that is a value can be used.

  For example, when the average value V2 of the brightness values V of the pixels included in the peripheral area of the detection area is 60 or less, the third determination unit 68 determines that the detection area is not a part corresponding to a foreign object. . This is because, when the brightness of the area around the detection area is lower than a predetermined value, even if a foreign object is reflected in the detection area, it is considered that the influence of the reflection of the foreign object on the image is small.

  The fourth determination unit 70 determines that the detection area to be determined is not a part corresponding to a foreign object when the standard deviation of brightness in the area around the detection area is not in a predetermined range. The fourth determination unit 70 determines that the detection area to be determined is not a part corresponding to a foreign object when the standard deviation of brightness in the area around the detection area is a predetermined range. To do. Here, as the standard deviation of the brightness in the area around the detection area, the standard deviation of the brightness value V of the pixel in the area around the detection area and the standard deviation of the luminance value Y of the pixel in the area around the detection area Is mentioned.

  For example, the fourth determination unit 70 calculates the brightness value V (i, j) of pixels included in the area around the detection area and the brightness value V (i, j) of pixels included in the area around the detection area. From the average value V2, the standard deviation Vstd of the brightness in the area around the detection area is calculated using the following equation (5).

  Note that n in Expression (5) is the number of pixels included in the area around the detection area. In the fourth determination unit 70, a threshold value for determining whether or not a foreign object is reflected in the detection area is appropriately set depending on the shooting condition of the shot image. For example, when the standard deviation Vstd of the brightness in the area around the detection area is 0.6 or more, the fourth determination unit 70 does not correspond to the foreign object reflected in the image. Can be judged. If the standard deviation of brightness in the surrounding area of the detection area is large, it is likely that the high brightness gradient value or luminance gradient value in the detection area is due to subject light, not due to reflection of foreign matter. It is.

  The fourth determination unit 70 may calculate the standard deviation of the brightness in the area around the detection area from the luminance value Y of the pixels included in the area around the detection area. Even in the case where the standard deviation calculated from the luminance information is used, the fourth determination unit 70 is a portion corresponding to the foreign object in the detection area to be determined as in the case where the standard deviation is calculated from the brightness information. It can be determined whether or not.

  As described above, the first to fourth determination units 64 to 70 included in the foreign object determination unit 48 independently determine whether or not the detection region to be determined is a part corresponding to the foreign object reflected in the image. Can be determined. The foreign matter determination unit 48 can output all the determination results of the first to fourth determination units 64 to 70 to the detection correction unit 44 and the foreign matter image correction unit 50. Further, the foreign matter determination unit 48 comprehensively determines whether or not a foreign matter is reflected in each detection region using the determination results of the first to fourth determination units 64 to 70, and determines the comprehensive determination result. Alternatively, the data may be output to the detection correction unit 44 and the foreign object image correction unit 50.

  For example, when one or more of the first to fourth determination units 64 to 70 determine that the detection area to be determined is not a part corresponding to the foreign object, the foreign object determination unit 48 is a determination target. A comprehensive determination is made that the detected area is not a part corresponding to a foreign object. Alternatively, the foreign matter determination unit 48 becomes a determination target when all the first to fourth determination units 64 to 70 have determined that the detection area to be determined is a portion corresponding to a foreign object. A comprehensive determination is made that the detected area is a portion corresponding to a foreign object.

  The foreign object image correcting unit 50 reduces the influence of the foreign object on the part corresponding to the foreign object reflected in the photographed image using the determination result of the foreign object determining unit 48 and the data calculated by the information calculating unit 46. Make corrections. The foreign object image correcting unit 50 performs correction to reduce the influence of the foreign object from the captured image based on the correction result that is the determination result corrected by the detection correcting unit 44 instead of the determination result in the foreign object determining unit 48. May be.

  The detection correction unit 44 can correct the determination of the foreign matter determination unit 48 in accordance with the operation signal input to the calculation unit 38 via the operation signal input unit 32. For example, the detection correction unit 44 changes at least one of the threshold values used in the first to fourth determination units 64 to 70 of the foreign object determination unit 48 according to the operation signal. Thereby, the detection correction part 44 can judge again whether the detection area | region is a part corresponding to a foreign material, and can correct the judgment of the foreign material determination part 48. FIG.

  FIG. 5 is a conceptual diagram showing the correction process of determination by the detection correction unit 44. FIG. 5 is a plot of the detection area calculated by the area calculation unit 62 in accordance with the size of the detection area and the lightness ratio inside and outside the detection area (the ratio between the first lightness information and the second lightness information). . Each of the circle mark 98 and the triangle mark 96 in FIG. 5 corresponds to one detection area. Note that the circle 98 corresponds to a detection region in which the detection correction unit 44 re-determines that the portion corresponds to a foreign object. Further, the triangle mark 96 corresponds to a detection area in which the detection correction unit 44 re-determines that the portion does not correspond to a foreign object.

  In the example illustrated in FIG. 5, the detection correction unit 44 illustrated in FIG. 2 performs the determination performed by the first determination unit 64 (determination based on the size of the detection region) and the determination performed by the second determination unit 66 (detection). Correction based on the difference between the brightness of the area and the brightness of the area around the detection area). The detection correction unit 44 can change the threshold value C1 related to the size of the detection region and the threshold value C2 related to the brightness ratio inside and outside the detection region based on the operation signal. This is because the threshold value C1 relating to the size of the detection region and the threshold value C2 relating to the brightness ratio inside and outside the detection region have a great influence on the detection sensitivity.

  Here, in the detection region represented by the triangle mark 96 and the circle mark 98 in FIG. 5, the portion corresponding to the foreign substance that has a more significant effect on the photographed image as it is plotted in the upper left direction indicated by the arrow 111. It is thought that. This is because a detection area having a larger detection area and a larger lightness difference with respect to the surrounding area (lightness ratio (first lightness information / second lightness information) is smaller) is more conspicuous. On the contrary, it is considered that the detection region plotted in the lower right direction indicated by the arrow 113 has less influence on the captured image. This is because the smaller the detection area determined to be a foreign object and the smaller the difference in brightness with respect to the surrounding area, the less noticeable.

  The detection correction unit 44 shown in FIG. 2 changes only the detection region larger than the threshold C1 in the part corresponding to the foreign matter by changing the threshold C1 shown in FIG. It is possible to re-determine that there is a foreign object and reduce the false detection of foreign matter. On the other hand, the detection correction unit 44 can reduce the leakage of foreign matter by changing the threshold value C1 shown in FIG. 5 in the direction of decreasing according to the operation signal from the input unit 32. Moreover, the detection correction part 44 can adjust detection sensitivity similarly to the case of the threshold value C1, by increasing / decreasing the threshold value C2 according to an operation signal.

  The detection correction unit 44 may change the values of the threshold C1 and the threshold C2 independently. For example, a predetermined function (such as a linear function) having the threshold C1 and the threshold C2 as variables is used to detect the threshold C1 and the threshold C2. May be changed according to one parameter. The threshold value that is changed by the detection correction unit 44 is not limited to the threshold value C1 related to the size of the detection region and the threshold value C2 related to the brightness ratio inside and outside the detection region.

  For example, the threshold value changed by the detection correction unit 44 is the threshold value related to the saturation value S of the second determination unit 66, the threshold value related to the luminance value Y of the second determination unit 66, or the threshold value related to the hue value H of the second determination unit 66. Alternatively, it may be a threshold value related to the brightness value V of the surrounding area in the third determination unit 68 or a threshold value related to the standard deviation Vstd of the brightness of the surrounding area in the fourth determination unit 70. Further, even if the detection correction unit 44 changes the calculation result and the determination result in the information calculation unit 46 and the foreign matter determination unit 48 by changing the threshold value regarding the lightness gradient value Vgrad (i, j) in the region calculation unit 62. Good. The detection correction unit 44 corrects the determination result in the foreign matter determination unit 48 by changing these threshold values. Thereby, the image processing apparatus according to the present embodiment can perform foreign object detection according to the user's preference.

  The detection correction unit 44 superimposes the captured image on the display unit 36 to display the determination result of the foreign matter determination unit 48 and the position of the detection area that is again determined by the detection correction unit 44 as a part corresponding to the foreign matter. You may let them. Accordingly, the user of the camera 10 can cause the detection / correction unit 44 to perform determination correction according to preference while visually recognizing the foreign substance detection sensitivity of the image processing apparatus. In addition, the detection correction unit 44 receives an operation signal for directly determining the position of the detection area in the photographed image and correcting the determination as to whether or not the detection area is a part corresponding to a foreign object. Also good. Thereby, the detection correction part 44 can perform judgment correction according to a user's liking more.

  Further, the detection / correction unit 44 stores a correction determination criterion, which is a corrected determination criterion, and data (foreign matter map) in which the determination of the detection / correction unit 44 is associated with the data of the detection region as shown in FIG. May be output. The correction criterion includes the above-described threshold C1 and threshold C2. Further, the data in which the determination result of the detection correction unit 44 is associated with the data of the detection area includes the classification of the detection area, the size of the detection area and the value of the brightness ratio, and the position of the detection area as shown in FIG. It is. The storage unit 30 can store the correction determination criteria corrected by the detection correction unit 44 and the data of the detection area, and can output them to the determination criterion change unit 42 as necessary.

  The determination criterion changing unit 42 can change the initial determination criterion, which is a determination criterion when the first to fourth determination units 64 to 68 in the foreign matter determination unit 48 make a determination, based on the correction determination criterion. The determination criterion changing unit 42 can change the initial determination criterion of the foreign matter determination unit 48 so as to approach the correction determination criterion.

  The initial determination criterion is a determination criterion related to the determination performed by the foreign matter determination unit 48 before the correction by the detection correction unit 44. The correction determination criterion is a determination criterion used in the correction determination by the detection correction unit 44. Therefore, the determination criterion changing unit 42 can bring the initial determination criterion of the foreign matter determination unit 48 closer to the preference of the user of the camera 10. Thereby, the image processing apparatus according to the present embodiment can efficiently perform foreign object detection according to the user's preference. The determination criterion changing unit 42 calculates an average value of a plurality of correction determination criteria stored in the storage unit 30, and changes the initial determination criterion of the foreign matter determination unit 48 based on the average value of the correction determination criteria. May be.

  In addition, the determination criterion changing unit 42 sets the initial determination criterion of the foreign matter determination unit 48 by using the accumulated data obtained by accumulating the detection region data corresponding to the determination result of the detection correction unit 44 over a plurality of determinations. It may be changed. For example, the determination criterion changing unit 42 determines that the ratio of the detection area (circle 98 in FIG. 5) determined to be a part corresponding to a foreign object is not a part corresponding to the foreign object (FIG. 5). A range larger than the ratio of the triangle mark 96) is calculated from the accumulated data of the foreign matter map. Then, the determination criterion changing unit 42 changes the initial determination criterion of the foreign matter determination unit 48 based on the range calculated from the accumulated data of the foreign matter map. Thereby, the determination criterion changing unit 42 can bring the initial determination criterion of the foreign matter determination unit 48 closer to the preference of the user of the camera 10.

  The foreign object image correcting unit 50 is an image that reduces the influence of the foreign object on the part corresponding to the foreign object reflected in the captured image based on the determination by the foreign object determining unit 48 or the determination corrected by the detection correcting unit 44. Make corrections. The foreign object image correction unit 50 according to the present embodiment includes a first image correction unit 74, a second image correction unit 76, and a third image correction unit 78.

  The first image correction unit 74 uses the brightness information calculated by the brightness information calculation unit 54 to perform a first correction for reducing the influence of the foreign matter on the part corresponding to the foreign matter reflected in the captured image. It can be carried out. The first image correction unit 74 performs the first image correction using the brightness value V and the brightness plane of each pixel calculated by the brightness information calculation unit 54.

  The first image correction unit 74 performs a normalization (normalization) process on the local lightness value V, calculates a lightness transmittance signal VT (i, j) for the detection area to be corrected, and performs the correction target. A lightness transmittance map (lightness gain map) is generated for the detection region. Here, the detection region to be corrected in the first image correction unit 74 is a detection region that is determined by the foreign matter determination unit 48 or the detection correction unit 44 as a portion corresponding to the foreign matter.

  The lightness transmittance signal VT (i, j) is an average value of the lightness value V (i, j) of the pixel of interest (i, j) in the detection region and the lightness value of the local range including the pixel of interest (i, j). It is calculated by calculating the relative ratio to. The first image correction unit 74 in the present embodiment can calculate the lightness transmittance signal VT (i, j) by the following equation (6).

  In Expression (6), α and β correspond to a range that takes a local average. α represents the number of pixels extending left and right around the pixel of interest (i, j), and β represents the number of pixels extending up and down around the pixel of interest (i, j). In the calculation of the lightness transmittance signal VT (i, j), the range for taking the local average can be set to about three times the assumed foreign object reflection region, but is not particularly limited. For example, in the calculation of the lightness transmittance signal VT (i, j), the first image correction unit 74 can set the values of the pixel numbers α and β to α = 36 to 55 and β = 36 to 55.

  FIG. 4 is a conceptual diagram illustrating generation of a lightness transmittance map performed in the first image correction unit 74. FIG. 4A shows a change in the lightness value V in the left-right direction on the lightness plane of the photographed image. In the lightness plane (curve) 81 shown in FIG. 4A, there are dip portions 82 and 84 in which the lightness value V is lower than the peripheral portion and the lightness gradient value Vgrad (i, j) is large. In FIG. 4A, the detection areas to be corrected by the first image correction unit 74 are the detection areas A and B corresponding to the drop portions 82 and 84.

  The first image correction unit 74 illustrated in FIG. 2 performs the lightness transmittance signal VT for each pixel included in the detection area A of the lightness plane (curve) illustrated in FIG. 4A and each pixel included in the detection area B. (I, j) is calculated. FIG. 4B shows a lightness transmittance map calculated from the lightness plane (curve) shown in FIG. In the lightness planes shown in FIG. 4A, the drop portions 82 and 84 in the lightness plane are extracted only by the lightness reduction due to the foreign matter by the local normalization process. In the lightness transmittance map shown in FIG. Are replaced by the sagging portions 82a and 82b. In this way, the first image correction unit 74 generates a lightness transmittance map for the detection region determined to be a part corresponding to a foreign object. The first image correction unit 74 generates a gain map for each detection region, thereby calculating a first correction amount (lightness gain) that is a correction amount for the first image correction independently for each detection region. May be.

  Further, the first image correction unit 74 uses the position information of each detection region and the lightness transmittance map of each detection region to reduce or eliminate the influence of the foreign matter on the part corresponding to the foreign matter reflected in the captured image. The first image correction is performed. The first image correction unit 74 performs gain correction by multiplying the lightness value V (i, j) of each pixel included in the detection region by the reciprocal of the lightness transmittance signal VT (i, j). That is, the first image correction unit 74 performs gain correction on the brightness value V of the pixel included in the detection area corresponding to the foreign object reflected in the captured image, and reduces or eliminates the change in brightness due to the reflection of the foreign object. For example, the first image correction unit 74 calculates the corrected brightness value VC (i, j) using the following equation (7).

  In this way, the first image correction unit 74 replaces the brightness value V (i, j) of the detection area with the above-described gain-corrected brightness value VC (i, j), thereby changing the brightness plane of the captured image. to correct. Accordingly, the first image correction unit 74 can reduce or remove the change in brightness due to the reflection of foreign matter from the captured image.

  The second image correction unit 76 uses the saturation information calculated by the saturation information calculation unit 56 to reduce the influence of the foreign matter on the portion corresponding to the foreign matter reflected in the captured image. Correction can be performed. The second image correction unit 76 performs second image correction using the saturation value S and the saturation plane of each pixel calculated by the saturation information calculation unit 56.

  Similar to the first image correction unit 74, the second image correction unit 76 performs a normalization (normalization) process on the local saturation value S, and a saturation transmittance signal is detected for the detection region to be corrected. ST (i, j) is calculated, and a saturation transmittance map (saturation gain map) for the detection area to be corrected is generated. Like the first image correction unit 74, the detection region to be corrected by the second image correction unit 76 is a detection region that is determined to be a part corresponding to the foreign material by the foreign material determination unit 48 or the detection correction unit 44. is there.

  The equation for calculating the saturation transmittance signal ST (i, j) is the lightness transmittance signal VT (i, j) and the lightness value V (i, j) in Equation (6), and the saturation transmittance signal ST (i , J) and the saturation value S (i, j) can be used. Similarly to the first image correction unit 74, the second image correction unit 76 generates a saturation transmittance map for the detection area determined to be a part corresponding to a foreign object. The second image correction unit 76 generates a gain map for each detection region, thereby calculating a second correction amount (saturation gain) that is a correction amount for the second image correction independently for each detection region. May be.

  Similarly to the first image correction unit 74, the second image correction unit 76 uses the position information of each detection region and the saturation transmittance map of each detection region to correspond to a foreign object reflected in the captured image. The second image correction for reducing or eliminating the influence of the foreign matter is performed. The second image correction unit 76 performs gain correction by multiplying the saturation value S (i, j) of each pixel included in the detection region by the reciprocal of the saturation transmittance signal ST (i, j). For example, the second image correction unit 76 calculates the corrected saturation value SC (i, j) using the following equation (8).

As described above, the second image correction unit 76 replaces the saturation value S (i, j) of the detection region with the above-described gain-corrected saturation value SC (i, j), thereby correcting the saturation of the captured image. Correct the degree plane. As a result, the second image correction unit 76 can reduce or remove the change in saturation due to the reflection of foreign matter from the captured image.

  The third image correction unit 78 uses the hue information calculated by the hue information calculation unit 58 to perform the third image correction that reduces the influence of the foreign matter on the portion corresponding to the foreign matter reflected in the captured image. It can be carried out. The third image correction unit 78 performs third image correction using the hue value H and the hue plane of each pixel calculated by the hue information calculation unit 58.

  Similar to the first image correction unit 74, the third image correction unit 78 performs a normalization (normalization) process of the local hue value H, and the hue transmittance signal HT ( i, j) is calculated, and a hue transmittance map (hue gain map) for the detection area to be corrected is generated. The detection region to be corrected in the third image correction unit 78 is a detection region that is determined to be a part corresponding to the foreign material in the foreign matter determination unit 48 or the detection correction unit 44, as in the first image correction unit 74. is there.

  The calculation formula of the hue transmittance signal HT (i, j) is the lightness transmittance signal VT (i, j) and the lightness value V (i, j) in the equation (6), and the hue transmittance signal HT (i, j). ) And hue value H (i, j) can be used. Similarly to the first image correction unit 74, the third image correction unit 78 generates a hue transmittance map for the detection area determined to be a part corresponding to a foreign object. The third image correction unit 78 generates a gain map for each detection region, thereby independently calculating a third correction amount (hue gain) that is a correction amount for the third image correction for each detection region. May be.

  Similar to the first image correction unit 74, the third image correction unit 78 uses the positional information of each detection region and the hue transmittance map of each detection region for the part corresponding to the foreign matter reflected in the captured image. Then, the third image correction is performed to reduce or remove the influence of the foreign matter. The third image correction unit 78 performs gain correction by multiplying the hue value H (i, j) of each pixel included in the detection region by the reciprocal of the hue transmittance signal HT (i, j). For example, the third image correction unit 78 calculates the corrected hue value HC (i, j) using the following equation (9).

  Thus, the third image correction unit 78 replaces the hue value H (i, j) of the detection area with the above-described gain-corrected hue value HC (i, j), thereby changing the hue plane of the captured image. to correct. As a result, the third image correction unit 78 can reduce or remove the change in hue due to the reflection of foreign matter from the captured image.

For example, when the color correction determination unit 52 determines that the captured image that is currently being corrected is an image that has been subjected to nonlinear correction, such as a JPEG image, for example, HSV correction for correcting the captured image by the first to third image corrections by the third image correction units 74 to 78 is performed.
HSV correction by the first to third image correction units 74 to 78 performs HSV conversion on a captured image, separately creates a gain map for each of the lightness plane, the saturation plane, and the hue plane, and performs image correction. Therefore, the HSV correction by the first to third image correction units 74 to 78 is performed in such a manner that even if the color of the photographed image is subjected to nonlinear correction, the color change due to the reflection of the foreign matter is It is possible to correct the color tone while maintaining high consistency.

  When the color correction determination unit 52 determines that the captured image that is currently being corrected is an image that has not been subjected to nonlinear correction, such as a RAW image, for example, Correction based only on the luminance (or correction based only on the brightness) may be performed.

  For example, general luminance correction is performed as follows. A local luminance value Y is normalized (normalized) to calculate a luminance transmittance signal YT (i, j) for the detection region to be corrected, and the luminance transmission for the detection region to be corrected. A rate map (luminance gain map) is generated. The calculation formula of the luminance transmittance signal YT (i, j) is the brightness transmittance signal VT (i, j) and the brightness value V (i, j) in the equation (6), and the luminance transmittance signal YT (i, j). ) And the luminance value Y (i, j) can be used.

  Reciprocal of luminance transmittance signal YT (i, j) for each of R value R (i, j), G value G (i, j) and B value B (i, j) of each pixel included in the detection region The gain is corrected by multiplying by. For example, the corrected R value RC (i, j), G value GC (i, j), and B value BC (i, j) are calculated using the following equations (10-1) to (10-3). calculate.

このように、検出領域に含まれる各画素のＲＧＢ値を、上述のゲイン補正されたＲ値ＲＣ（ｉ，ｊ）、Ｇ値ＧＣ（ｉ，ｊ）およびＢ値ＢＣ（ｉ，ｊ）に置き換える（一般的な輝度のみによる補正）ことによって、撮影画像を補正してもよい。

In this way, the RGB value of each pixel included in the detection area is replaced with the above-described gain-corrected R value RC (i, j), G value GC (i, j), and B value BC (i, j). The captured image may be corrected by (correction using only general luminance).

  In general correction only by luminance, correction is performed using only the luminance gain map (YT (i, j)) as shown in equations (10-1) to (10-3). For this reason, if correction is performed only for general luminance on an image that has been nonlinearly corrected, the RGB values (RC (i, j), GC (i, j), BC (i, j) in the pixel after correction are corrected. )) Becomes a value far from the RGB values in the peripheral pixels, and correction marks may occur.

  Such a problem is likely to occur when a strong non-linear correction is applied to a captured image to be corrected. In addition, the non-linear correction method for the photographed image may differ depending on the model of the photographing apparatus such as a camera. On the other hand, in the HSV correction according to the present embodiment, as shown in the equations (7) to (9), the brightness gain map, the saturation gain map, and the hue gain map are respectively generated and corrected. The corrected pixel is corrected to a color close to that of the peripheral pixel. Further, according to the HSV correction, even when the captured image to be corrected is subjected to nonlinear correction, it is possible to prevent a problem that a correction mark is generated on the corrected captured image, and to correct the correction. There is no limitation on the form of nonlinear correction of the target image. That is, for example, the effect of the present embodiment can be expected for an image after nonlinear correction other than JPEG.

  Note that the HSV correction by the foreign object image correcting unit 50 may generate a corrected captured image by combining the lightness plane, the saturation plane, and the hue plane corrected by the first to third image correcting units 74 to 78. The image correction by the second image correction unit 74 or the third image correction unit 78 may be omitted. That is, in the HSV correction by the foreign object image correction unit 50, gain correction is not performed for either the saturation plane or the hue plane, and the saturation plane or hue plane that has not been gain-corrected is used as the corrected brightness plane or the like. By combining them, a corrected captured image may be generated.

  In the present embodiment, by combining correction by lightness using the first image correction unit 74 (including correction by luminance) with correction by saturation using the second image correction unit 76, only general luminance can be obtained. Compared with the correction, the accuracy of the foreign substance removal correction for an image that has been nonlinearly corrected is improved. Further, in the present embodiment, only general luminance is obtained by combining correction by brightness using the first image correction unit 74 (including correction by luminance) with correction by hue using the third image correction unit 78. Compared with this correction, the accuracy of the foreign matter removal correction for an image that has been nonlinearly corrected is improved.

  In the HSV correction, the foreign object image correction unit 50 may calculate the first to third correction amounts, which are correction amounts related to the first to third image corrections, independently for each detection region. By calculating the correction amount for each detection region, the image processing apparatus according to the present embodiment can suppress the calculation amount as compared with the case where the correction amount is calculated for the entire captured image. Of course, the first image correction unit of the foreign object image correction unit 50 may perform image correction using luminance information instead of brightness information.

  The color correction determination unit 52 applies to a captured image obtained by photoelectrically converting an image taken by the optical system 16 in the camera 10, which is a correction target for removing the influence of the foreign material in the foreign material image correcting unit 50. It is then determined whether or not nonlinear correction has been performed. The color correction determination unit 52 determines whether or not the captured image that is a correction target for removing the influence of the foreign matter in the foreign matter image correction unit 50 has been subjected to nonlinear correction, and the foreign matter image correction unit The determination result is output to 50.

  The foreign object image correcting unit 50 uses the first to third image correcting units 74 to 78 when the color correction determining unit 52 determines that the photographed image to be corrected has been subjected to nonlinear correction. The HSV correction described above is performed, and the foreign object image correction unit 50 performs correction to reduce the influence of the foreign object. On the contrary, the foreign object image correction unit 50 determines that the captured image to be corrected currently is not subjected to nonlinear correction by the color correction determination unit 52, and the first to third image correction units 74. The correction by only the general luminance may be performed without performing the image correction by ˜78, and the foreign object image correcting unit 50 may perform the correction to reduce the influence of the foreign object.

  In addition, the color correction determination unit 52 may determine the strength of nonlinear correction applied to the captured image that is a correction target for removing the influence of the foreign matter in the foreign matter image correction unit 50. The color correction determination unit 52 determines the level of nonlinear correction applied to the captured image that is a correction target for removing the influence of the foreign matter in the foreign matter image correction unit 50, and outputs the judgment result to the foreign matter image correction unit 50. To do.

  In this case, the foreign object image correction unit 50 determines that the first to third images when the captured image to be corrected is determined to have been subjected to strong nonlinear correction by the color correction determination unit 52. The HSV correction described above is performed by the correction units 74 to 78, and the foreign image correction unit 50 performs correction to reduce the influence of the foreign material. On the contrary, the foreign object image correction unit 50 performs the first to third image corrections when the color correction determination unit 52 determines that the captured image to be corrected has been subjected to weak nonlinear correction. The correction by only the general luminance may be performed without performing the image correction by the unit 78, and the foreign image correction unit 50 may perform the correction to reduce the influence of the foreign material.

  The storage unit 30 stores correction determination criteria corrected by the detection correction unit 44, detection area data, shooting conditions of captured images, and the like. The storage unit 30 is configured by a nonvolatile memory or the like. The operation signal input unit 32 is an input unit to which an operation signal corresponding to an operation performed by the user in order to specify a part corresponding to a foreign object reflected in a captured image. The operation signal input unit 32 is used by the first to fourth determination units 64 to 70 of the foreign object determination unit 48 such as a threshold value C1 regarding the size of the detection region and a threshold value C2 regarding the brightness ratio inside and outside the detection region. An operation signal for changing the threshold value (judgment criterion) is input.

  The operation signal input unit 32 is a user interface for a user to input an operation signal to the image processing apparatus, such as a button, a cross key, and a rotary selector provided in the camera body unit 12 (see FIG. 1). Including. The detection correction unit 44 can correct the determination of the foreign matter determination unit 48 in accordance with the operation signal input to the calculation unit 38 via the operation signal input unit 32.

  The imaging condition signal input unit 34 is input with imaging conditions relating to a captured image determined by the foreign matter determination unit 48 and a captured image on which image correction is performed by the foreign matter image correction unit 50. The shooting conditions input to the shooting condition signal input unit 34 include, for example, the aperture value and shooting sensitivity when shooting a shot image, the type of optical system used when shooting the shot image, the focal length, and the like.

  The detection / correction unit 44 stores a correction determination criterion, which is a correction criterion corrected according to the imaging condition input via the imaging condition signal input unit 34, and data relating to the foreign matter map as shown in FIG. 30 can be stored. In addition, the determination criterion changing unit 42 is an initial determination criterion that is a criterion for determination by the first to fourth determination units 64 to 70 in the foreign matter determination unit 48 based on the correction determination criterion corresponding to the shooting condition. Can be changed for each shooting condition. The foreign object detection sensitivity according to the user's preference may change depending on the shooting conditions, and the image processing apparatus including the shooting condition signal input unit 34 uses the initial determination criterion of the foreign object determination unit 48 as a reference of the camera 10. The user's preference can be approached.

  Further, in the case where foreign matter at a specific position of the optical filter 20 continues to adhere, if the photographing conditions are the same, there is a high possibility that the foreign matter will continue to appear in a predetermined position of the photographed image. Therefore, by storing the data related to the foreign matter map according to the shooting conditions, the image processing apparatus according to the present embodiment can easily specify the reflected position of the foreign matter that the user is interested in.

  The display unit 36 is configured by, for example, a liquid crystal display device provided in the camera body 12 (see FIG. 1). The display unit 36 can display the position of the detection region that is re-determined as a part corresponding to the foreign substance by the detection correction unit 44 so as to overlap the captured image.

  3A and 3B are flowcharts illustrating an example of the image processing calculation performed in the camera 10 illustrated in FIG. FIG. 3A shows the first half of the image processing calculation, and FIG. 3B shows the second half of the image processing calculation. As shown in FIGS. 3A and 3B, the image processing calculation includes calculation I including steps S003 to S007, calculation II including steps S008 to S011, calculation III including steps S012 to S016, and steps. And an operation IV including S017 to step S020.

  In the calculation I, the calculation unit 38 performs a calculation for calculating information regarding the color of the captured image, such as brightness information and saturation information. In the calculation II, the calculation unit 38 uses the information calculated in the calculation I to calculate foreign matter information related to the foreign matter reflected in the captured image, such as a position where the foreign matter is reflected. In the calculation II, the calculation unit 38 uses the information calculated in the calculation I to determine whether or not the detection area in the captured image is a part corresponding to a foreign object reflected in the captured image. .

  In the calculation III, the calculation unit 38 inputs a user operation signal via the operation signal input unit 32, and performs a calculation to correct the determination made in the calculation II in accordance with the input operation signal. In the calculation IV, the calculation unit 38 specifies the detection area in which the foreign object is reflected based on the determination result in the calculation II or the calculation III, and then uses the information calculated in the calculation I to display the captured image. Image correction for reducing the influence of the foreign matter is performed on the portion corresponding to the foreign matter. In the calculation IV, the calculation unit 38 performs the first image correction (image correction by the first image correction unit 74) that reduces the influence of the foreign matter using the brightness information calculated in the calculation I. Further, in the calculation IV, the second image correction (image correction by the second image correction unit 76) for reducing the influence of the foreign matter using the saturation information calculated in the calculation I, and the hue information calculated in the calculation I are used. The third image correction (image correction by the third image correction unit 78) is used to reduce the influence of the foreign matter.

  The image processing apparatus mounted on the camera 10 shown in FIG. 2 starts a series of image processing operations in step S001 as shown in FIG. 3A. In step S <b> 002, the captured image captured by the imaging unit 26 is input to the calculation unit 38 via the image input unit 28.

  FIG. 6 illustrates an image processing calculation performed in the camera 10 illustrated in FIG. 2 by taking a captured image b (reference symbol b), which is a part of the captured image indicated by the symbol a, as an example. In step S002 illustrated in FIG. 3A, the captured image b is input to the calculation unit 38.

  In steps S003 to S007 shown in FIG. 3A, the information calculation unit 46 in the calculation unit 38 shown in FIG. 2 calculates information related to the color of the captured image. In step S003, the brightness information calculation unit 54 of the information calculation unit 46 calculates brightness information corresponding to the brightness of the captured image. The brightness information calculation unit 54 may calculate the luminance plane of the captured image or the brightness plane of the captured image. In step S003 shown in FIG. 3A, the brightness information calculation unit 54 The brightness plane is calculated using the above equations (2-1) and (2-6).

  In step S003, the saturation information calculation unit 56 of the information calculation unit 46 calculates the saturation plane using the above-described equations (2-2), (2-6), and (2-7). In step S005, the hue plane is calculated using the above-described equations (2-3) to (2-5) and (2-6) to (2-10).

  As shown in FIG. 6, in step S003, the brightness plane d (symbol d) of the captured image b is calculated from the captured image b. In step S004, a saturation plane (not shown) of the captured image b is calculated from the captured image b. In step S005, a hue plane c of the captured image b is calculated from the captured image b.

  In step S006 shown in FIG. 3A, the brightness gradient information calculation unit 60 of the information calculation unit 46 calculates brightness gradient information corresponding to the brightness gradient of the captured image. The brightness gradient information calculation unit 60 may calculate the brightness gradient plane of the captured image or the brightness gradient plane of the captured image. In step S006 shown in FIG. 3A, the brightness gradient information calculation is performed. The unit 60 calculates the lightness gradient plane using the lightness plane data calculated in step S003 and the above-described equation (4).

  In step S007 illustrated in FIG. 3A, the area calculation unit 62 of the information calculation unit 46 calculates a detection area. The area calculation unit 62 calculates a binarized image using the brightness gradient plane calculated by the brightness gradient information calculation unit 60, and calculates a detection area. The region calculation unit 62 identifies a region where pixels having a lightness gradient value higher than the threshold are connected as one detection region.

  As shown in FIG. 6, in step S006, the brightness gradient plane e (symbol e) of the captured image b is calculated from the brightness plane d (symbol d) of the captured image b. In step S007, a binarized image f (symbol f) is calculated from the lightness gradient plane e.

  In step S008 to step S011 shown in FIG. 3A, the foreign matter determination unit 48 in the calculation unit 38 shown in FIG. 2 uses the information calculated by the information calculation unit 46 for each detection region specified by the region calculation unit 62. Then, it is determined whether or not the detection area is a part corresponding to the foreign matter.

  In step S008 shown in FIG. 3A, the first determination unit 64 of the foreign object determination unit 48 corresponds to the foreign object reflected in the image based on the size of the detection area to be determined. It is determined whether it is a part. The first determination unit 64 can calculate the size of the detection region from the number of pixels included in the detection region using the binarized image f. Further, if the size of the detection area calculated based on the binarized image f is not within a predetermined range, the first determination unit 64 does not correspond to the foreign object. Judge.

  In step S009 shown in FIG. 3A, the second determination unit 66 of the foreign matter determination unit 48 corresponds to the saturation information corresponding to the saturation of the detection region, the hue information corresponding to the hue of the detection region, and the brightness of the detection region. Using the brightness information to be determined, it is determined whether or not the detection area is a portion corresponding to a foreign object reflected in the image.

  In step S009, first, the second determination unit 66 uses the brightness plane calculated by the brightness information calculation unit 54, and the first brightness information corresponding to the brightness of the detection area and the brightness of the area around the detection area. The difference from the second lightness information corresponding to is investigated. In step S009, the second determination unit 66 uses the average value of the brightness values V of the pixels included in the detection area as the first brightness information, and determines the average value of the brightness values V of the pixels included in the surrounding area of the detection area. 2 brightness information. When the ratio between the first lightness information and the second lightness information is not within a predetermined range, the second determination unit 66 determines that the detection area to be determined is not a part corresponding to a foreign object.

  Next, the second determination unit 66 uses the saturation plane calculated by the saturation information calculation unit 56, and the first saturation information corresponding to the saturation of the detection region and the saturation of the region around the detection region. The difference from the second saturation information corresponding to the degree is investigated. In step S009, the second determination unit uses the average value of the saturation values S of the pixels included in the detection area as the first saturation information, and the average value of the saturation values S of the pixels included in the surrounding area of the detection area. Is the second saturation information. When the ratio between the first saturation information and the second saturation information is not within a predetermined range, the second determination unit 66 determines that the detection area to be determined is not a part corresponding to a foreign object. To do.

  Furthermore, the second determination unit 66 uses the hue plane calculated by the hue information calculation unit 58 to use the first hue information corresponding to the hue of the detection region and the second hue corresponding to the hue of the region around the detection region. Investigate differences from hue information. In step S009, the second determination unit 66 uses the average value of the hue values H of the pixels included in the detection area as the first hue information, and determines the average value of the hue values S of the pixels included in the area around the detection area as the first value. Two hue information is used. When the ratio between the first hue information and the second hue information is not within a predetermined range, the second determination unit 66 determines that the detection area to be determined is not a part corresponding to a foreign object.

  In step S010 shown in FIG. 3A, the third determination unit 68 of the foreign object determination unit 48 uses the brightness information corresponding to the brightness of the area around the detection area, and the detection area is reflected in the image. It is determined whether or not the portion corresponds to. The third determination unit 68 uses the brightness plane calculated by the brightness information calculation unit 54 to calculate second brightness information corresponding to the brightness of the area around the detection area. Similar to the second determination unit 66, the third determination unit 68 uses the average value of the lightness values V of the pixels included in the region around the detection region as the second lightness information. When the second brightness information is not in a predetermined range, the third determination unit 68 determines that the detection area that is the object of determination is not a part corresponding to a foreign object.

  In step S011 shown in FIG. 3A, the fourth determination unit 70 of the foreign object determination unit 48 determines that the detection area corresponds to the foreign object reflected in the image from the standard deviation of the brightness of the area around the detection area. Judge whether there is. The fourth determination unit 70 uses the brightness plane calculated by the brightness information calculation unit 54 and the above-described formula (5) to calculate the standard deviation of the brightness of the area around the detection area. The fourth determination unit 70 determines that the detection area to be determined is not a part corresponding to a foreign object when the standard deviation of the brightness of the area around the detection area is not in a predetermined range.

  As shown in FIG. 6, in steps S008 to S0012, a detection area that does not correspond to a foreign object is excluded from the binarized image f as in the binarized image g indicated by the symbol g, and the foreign object is reflected. Only the detection area is extracted.

  Note that the determinations corresponding to steps S008 to S011 in FIG. 3A may be made for all detection regions included in the captured image b, but the determinations corresponding to steps S009 to S011 are the previous steps S009 to step S011. You may perform only about the area | region which was not judged as the part corresponding to a foreign material in S011. The foreign matter determination unit 48 determines that only the detection region that is not determined to be a portion corresponding to the foreign matter in any of steps S008 to S011 is a portion corresponding to the foreign matter.

  In addition, the foreign matter determination unit 48 may sequentially determine from step S008 to step S011 for one detection area, and then sequentially determine from step S008 to step S011 for the other detection area. . In this case, the determination regarding one detection area is performed when it is determined in any one of the steps included in steps S008 to S011 that the detection area to be determined is not a part corresponding to a foreign object. The remaining determination (step) for the region may be skipped, and the determination for another detection region may be started. In this case, the foreign matter determination unit 48 determines that the detection region that is a determination target in step S011 and that is not determined to be a portion corresponding to the foreign matter in step S011 is a portion corresponding to the foreign matter. to decide. By enabling the foreign object determination unit 48 to skip the determinations corresponding to steps S009 to S011, the image processing apparatus can reduce the amount of calculation in the foreign object determination unit 48.

  The calculation unit 38 performs the calculation III including the steps S012 to S016 in FIG. 3B after completing the calculation II including the steps S007 to S011 in FIG. 3A. The calculation unit 38 does not perform the calculation III after completing the calculation II including the steps S007 to S011 in FIG. 3A according to the setting of the user and the like, and performs the calculation including the steps S017 to S020 in FIG. 3B. IV may be performed.

  In step S012 to step S016 shown in FIG. 3B, the correction determination unit 44 of the calculation unit 38 inputs a user operation signal via the operation signal input unit 32, and steps S008 to S0 are performed according to the input operation signal. An operation for correcting the determination made in step S011 is performed.

  In step S012 shown in FIG. 3B, the detection / correction unit 44 causes the display unit 36 to display the determination result of the foreign matter determination unit 48 on the captured image. As shown in the image indicated by symbol h in FIG. 6, the display unit 36 receives an instruction from the detection / correction unit 44 and is superposed on the captured image b and is determined to be a part corresponding to the foreign object by the foreign object determination unit 48. The detected area is displayed separately from other detected areas. In the image indicated by the symbol h, only the position of the detection region that is determined to be a part corresponding to the foreign material by the foreign material determination unit 48 among the detection regions extracted by the region calculation unit 62 uses a triangular figure. It is displayed.

  In step S013 shown in FIG. 3B, an operation signal for correcting the determination made by the foreign matter determination unit 48 is input to the detection correction unit 44 of the calculation unit 38 via the operation signal input unit 32. In step S013, the user operates the user interface included in the operation signal input unit 32 while correcting the determination result displayed on the display unit 36 to correct the determination made by the foreign matter determination unit 48. The operation signal can be input.

  When the user considers that the foreign matter detection unit 48 determines that the foreign matter detection sensitivity is too low, the user uses the judgment so that the determination is performed with a higher detection sensitivity than the judgment made by the foreign matter judgment unit 48. An operation signal for changing the threshold value is input. For example, the user increases an operation signal to be changed so as to decrease the threshold value C1 related to the size of the detection region (see FIG. 6) or a threshold value C2 related to the brightness ratio inside and outside the detection region (see FIG. 6). Input the operation signal to be changed.

  On the other hand, if the user thinks that the foreign matter determination unit 48 determines that the foreign matter detection sensitivity is too high, the user can make a re-determination with a lower detection sensitivity than the determination made by the foreign matter determination unit 48. An operation signal for changing a threshold value used for determination is input. For example, the user decreases the operation signal to be changed so as to increase the threshold value C1 (see FIG. 6) related to the size of the detection region, or the threshold value C2 (see FIG. 6) related to the brightness ratio inside and outside the detection region (to 0). Input the operation signal to be changed.

  In step S014 shown in FIG. 3B, the detection correction unit 44 changes the threshold value used in the determination in the foreign matter determination unit 48 in accordance with the operation signal input to the calculation unit 38 via the operation signal input unit 32. It is determined again whether or not the detection area is a part corresponding to a foreign object. That is, in step S014, the detection correction unit 44 corrects the determination made by the first to fourth determination units 64 to 70 in the foreign matter determination unit 48 using the threshold value changed according to the operation signal.

  For example, in step S013, when an operation signal indicating that the threshold C1 (see FIG. 6) relating to the size of the detection area is to be reduced is input, the detection correction unit 44 performs the determination of the foreign matter determination unit 48 as follows. Correct it. That is, the detection correction unit 44 causes the foreign matter determination unit 48 to recalculate the calculation II including steps S008 to S011 after reducing the threshold C1 in the first determination unit 64 according to the operation signal. The detection correction unit 44 determines that the detection area determined to be a part corresponding to the foreign object by recalculation is the detection area re-determined to be the part corresponding to the foreign object.

  In step S015 shown in FIG. 3B, the detection correction unit 44 updates the display on the display unit 36. That is, based on the determination corrected by the detection correction unit 44, the display unit 36 displays the detection region that has been re-determined as a part corresponding to a foreign substance in step S014 over the captured image.

  The user confirms the display of the display unit 36 updated in step S015, and if the detection result re-determined by the detection correction unit 44 is considered appropriate, the re-determination of the detection area is terminated. An operation signal is input to the camera 10. In this case, the detection correction unit 44 proceeds with the process of step S016. In addition, when the user thinks that the detection result redetermined by the detection correction unit 44 is inappropriate, the user inputs an operation signal to redo the detection area to the detection correction unit 44. . In this case, the detection correction unit 44 redoes the processing according to steps S013 to S15.

  In step S016 shown in FIG. 3B, the detection correction unit 44 outputs to the storage unit 30 the threshold value at the time of re-determination, the position of the detection area determined to be a part corresponding to the foreign object in the re-determination. The storage unit 30 stores data such as a threshold value at the time of redetermination and a position of a detection region determined to be a part corresponding to a foreign substance, and stores the data in the determination criterion changing unit 42 as necessary. Data can be output.

  In step S017 to step S020 shown in FIG. 3B, the foreign object image correction unit 50 of the calculation unit 38 is a part corresponding to a foreign object by the detection correction unit 44 using the information calculated by the brightness information calculation unit 54. For the determined detection area, image correction is performed to reduce the influence of foreign matter from the captured image.

  In step S <b> 017, the foreign object image correction unit 50 identifies the detection area determined to be a part corresponding to the foreign object by the detection correction unit 44 as an image correction target area that reduces the influence of the foreign object from the captured image. . In step S018, the color correction determination unit 52 determines whether or not the captured image that is currently being corrected has been subjected to nonlinear correction by the image conversion unit 15 or the like.

  When it is determined in step S018 that the captured image has been subjected to nonlinear correction, in steps S019 and S020, image correction by the first to third image correction units 74 to 78 of the foreign object image correction unit 50 is performed. Done. In this case, in step S019, the first to third image correction units 74 to 78 of the foreign object image correction unit 50 are calculated by the brightness information calculated by the brightness information calculation unit 54 and the saturation information calculation unit 56. Using the saturation information and the hue information calculated by the hue information calculation unit 58, a gain map is generated for the image correction target region. The first to third image correcting units 74 to 78 of the foreign object image correcting unit 50 use the information calculated by the information calculating unit 46 and the above-described equation (6), and the lightness transmittance map, the saturation transmittance map, and Generate a hue transmittance map.

  In step S020, the first to third image correction units 74 to 78 of the foreign object image correction unit 50 obtain the lightness transmittance map, the saturation transmittance map, the hue transmittance map, and the above-described equations (7) to (9). The gain correction is performed on the brightness plane, the saturation plane, and the hue plane of the captured image. Further, the foreign object image correction unit 50 generates a corrected captured image by combining the brightness plane, the saturation plane, and the hue plane that have been gain-corrected by the first to third image correction units 74 to 78.

  When it is determined in step S018 that the captured image is not subjected to non-linear correction, in step S019 and step S020, correction using only general luminance may be performed.

  In step S021, after the corrected captured image generated in step S020 is stored, a series of image processing operations are terminated.

  As described above, the image processing apparatus according to the present embodiment uses the brightness information of the detection area and the saturation information of the detection area to determine whether the detection area corresponds to a foreign object reflected in the captured image. It has a foreign matter judgment unit 48 for judging whether or not. Since the foreign matter determination unit 48 determines not only the brightness information of the detection region but also the saturation information, the foreign matter determination unit 48 can accurately determine whether or not a foreign matter is reflected in the detection region. This is because the foreign matter reflected in the photographed image has a feature that it hardly affects the saturation and hue, but strongly affects the brightness. Moreover, in order to detect a color change due to the reflection of a foreign object from a photographed image, it is effective to make a determination by combining saturation information and brightness information.

  In addition to the brightness information of the detection area and the saturation information of the detection area, the image processing apparatus according to this embodiment determines the detection area using the hue information of the detection area. It is possible to more accurately determine whether or not the image is reflected. In addition, the image processing apparatus according to the present embodiment makes it possible to determine whether a foreign object is present in the detection area by determining the brightness information of the area around the detection area or the standard deviation of the brightness in the area around the detection area. It is possible to more accurately determine whether or not the image is reflected. This is because it is easy to mistakenly detect that the subject edge is an edge caused by the movement of a foreign object in a dark portion or a portion with a large variation in brightness in the captured image. Further, in these regions, even if there is an edge due to the transfer of foreign matter, it is relatively inconspicuous. In addition, the image processing apparatus according to the present embodiment performs the determination based on the brightness information using the saturation information or the hue information by performing the determination based on the saturation information or the hue information after performing the determination based on the brightness information. Can be supplemented or verified.

  The image processing apparatus according to the present embodiment includes a detection correction unit 44 that corrects the determination of the foreign matter determination unit 48 based on an operation signal from the user. Therefore, the image processing apparatus according to the present embodiment can detect a foreign object reflected in a captured image at a detection level that matches the user's preference. In addition, since the display unit 36 displays the detection results before and after correction by the detection / correction unit 44, the user adjusts the detection level while checking the display on the display unit 36, thereby preventing erroneous detection or omission of foreign matter. Can be prevented.

  Further, the image processing apparatus according to the present embodiment is based on the correction determination criterion by the detection correction unit 44, and is an initial determination that is a determination criterion when the first to fourth determination units 64 to 68 in the foreign matter determination unit 48 make a determination. A judgment standard changing unit 42 for changing the standard is provided. Since the determination criterion changing unit 42 can bring the initial determination criterion of the foreign matter determination unit 48 closer to the user's preference of the camera 10, the image processing apparatus according to the present embodiment can detect foreign matter according to the user's preference. Can be performed efficiently. The detection correction unit 44 may correct the threshold value when the region calculation unit 62 generates a binarized image, and the determination criterion change unit 42 determines whether the region calculation unit 62 generates a binarized image. The threshold value may be changed. The image processing apparatus according to the present embodiment can also optimize the foreign matter detection sensitivity by correcting the threshold when the region calculation unit 62 generates the binarized image or feeding back the threshold.

  The image processing apparatus according to the present embodiment adds the gain correction using the saturation gain map and / or the gain correction using the hue gain map to the gain correction using the brightness gain map, thereby reducing the influence of foreign matter reflected in the captured image. It has a foreign object image correction unit 50 that performs image correction to be reduced. Therefore, the image processing apparatus according to the present embodiment has a high color change due to the reflection of a foreign object and the hue of the peripheral portion even when the captured image to be corrected is subjected to nonlinear correction. Corrections can be made while maintaining consistency. In addition, the image processing apparatus according to the present embodiment prevents the problem that correction marks are generated in the corrected captured image even when the captured image to be corrected is subjected to nonlinear correction. it can.

  In the above-described embodiment, the imaging apparatus has been described by taking the camera 10 equipped with the image processing apparatus as an example. However, the imaging apparatus having the image processing apparatus is not limited to a still camera, and a video camera, Includes mobile phones. The program for executing the above-described image processing is not limited to a program for executing a computer mounted on a still camera, a video camera, or a mobile phone, but includes a program for executing a computer capable of performing other image processing. .

Second Embodiment FIG. 7 is a block diagram of a computer 120 such as a personal computer including an image processing apparatus according to a second embodiment of the present invention. The computer 120 does not include the imaging unit 26, the operation signal input unit 32, the imaging condition signal input unit 23, and the image conversion unit 115 illustrated in FIG. 2, but instead, as illustrated in FIG. 7, an I / F (interface). 122 and an operation signal input unit 124. The computer 120 shown in FIG. 7 has the same configuration as that of the camera 10 shown in FIG. 2 except for the imaging unit 26 and the like shown in FIG. The same reference numerals as in FIG.

  The computer 120 has an I / F 122 for connecting to a memory card, a camera, a mobile phone, or the like. A captured image is input to the image input unit 28 of the computer 120 from the I / F 122 or the storage unit 30 built in the computer 120. The calculation unit 38 illustrated in FIG. 7 performs image processing on the photographed image input via the image input unit 28, similarly to the calculation unit 38 in the camera 10 illustrated in FIG. It should be noted that shooting conditions and the like may be input to the calculation unit 38 along with the shot image via the I / F 122 or the like. The calculation unit 38 may be a dedicated circuit that constitutes the information calculation unit 46, the foreign matter determination unit 48, the foreign matter image correction unit 50, and the like existing therein, but preferably a program for realizing these functions. And a central information processing device (CPU) that executes a program stored in the storage means.

  Similar to the operation signal input unit 32 according to the first embodiment, the operation signal input unit 124 receives an operation signal corresponding to an operation performed by the user in order to specify a part corresponding to the foreign object reflected in the captured image. An input unit to be input. Examples of the operation signal input unit 124 include a keyboard, a mouse, and a touch pad, but are not particularly limited.

  As with the calculation unit 28 according to the first embodiment, the calculation unit 38 of the computer 120 detects a foreign object reflected in the shot image, and displays the shot image input through the image input unit 28 in the shot image. Image correction can be performed to reduce the influence of the foreign matter. Therefore, the image processing device included in the computer 120 also has the same effect as the image processing device mounted on the camera 10 according to the first embodiment.

DESCRIPTION OF SYMBOLS 10 ... Camera 26 ... Image pick-up part 28 ... Image input part 32 ... Operation signal input part 34,124 ... Operation condition signal input part 36 ... Display part 38 ... Calculation part 42 ... Judgment reference change part 44 ... Detection correction part 46 ... Information calculation Unit 48 ... Foreign matter determination unit 50 ... Foreign matter image correction unit 52 ... Color correction determination unit 54 ... Brightness information calculation unit 56 ... Saturation information calculation unit 58 ... Hue information calculation unit 60 ... Brightness gradient information calculation unit 62 ... Region detection Units 64-70 ... First to fourth determination units 74-78 ... First to third image correction units 115 ... Image conversion unit 120 ... Computer

Claims (8)

An image input unit for inputting a captured image;
After determining that the difference between the brightness of the detection area in the captured image and the brightness of the area around the detection area is greater than a first threshold, first saturation information corresponding to the saturation of the detection area; When the difference from the second saturation information corresponding to the saturation of the area around the detection area is determined to be smaller than the second threshold, the first determination is made to determine that the detection area is a part corresponding to a foreign object. And
A second determination unit that determines that the detection region is a portion corresponding to the foreign object when it is determined that a standard deviation of brightness of a region around the detection region is smaller than a third threshold;
An image processing apparatus comprising: a correction unit that corrects the detection area when the first determination unit and the second determination unit determine that the detection area corresponds to the foreign object. . An image processing apparatus according to claim 1,
The first determination unit is configured such that the difference between the brightness of the detection area and the brightness of the surrounding area of the detection area is not greater than the first threshold or corresponds to the saturation of the detection area. When it is determined that the difference between the information and the second saturation information corresponding to the saturation of the area around the detection area is not smaller than the second threshold , the detection area is not a part corresponding to the foreign matter An image processing apparatus characterized by determining . An image processing apparatus according to claim 1 or 2, wherein
The correction unit excludes the detection region from a correction target when the first determination unit or the second determination unit determines that the detection region is not a part corresponding to the foreign matter. apparatus. An image processing apparatus according to any one of claims 1 to 3, wherein
The first determination unit determines whether the detection region is a portion corresponding to the foreign object using a gradient between brightness of the detection region and brightness of a region around the detection region. An image processing apparatus. The image processing apparatus according to any one of claims 1 to 4, wherein:
Wherein the first determination unit uses said first chroma information corresponding to the saturation of the detection region, the ratio of the second chroma information corresponding to the saturation of the region around of the detection area An image processing apparatus that determines whether a detection area is a portion corresponding to the foreign matter. An image processing apparatus according to any one of claims 1 to 5, wherein
The first determination unit determines that the detection area is not a part corresponding to the foreign object when the second brightness information corresponding to the brightness of the area around the detection area is smaller than a fourth threshold value. A featured image processing apparatus. On the computer,
The procedure to input the shot image,
Calculating the difference between the brightness of the detection area in the captured image and the brightness of the area around the detection area ;
Calculating a difference between the first saturation information corresponding to the saturation of the detection region and the second saturation information corresponding to the saturation of a region around the detection region ;
Calculating the standard deviation of the brightness of the area around the detection area;
The first saturation information corresponding to the saturation of the detection area after determining that the difference between the brightness of the detection area in the captured image and the brightness of the area around the detection area is greater than a first threshold. And the second saturation information corresponding to the saturation of the area around the detection area is determined to be smaller than a second threshold value, it is determined that the detection area is a part corresponding to a foreign object. 1 judgment procedure;
A second determination procedure for determining that the detection area is a portion corresponding to the foreign object when it is determined that a standard deviation of the brightness of the area around the detection area is smaller than a third threshold;
A program for executing a procedure for correcting the detection area when it is determined by the first determination procedure and the second determination procedure that the detection area corresponds to the foreign matter . On the computer,
The procedure to input the shot image,
After determining that the difference between the brightness of the detection area in the captured image and the brightness of the area around the detection area is greater than a first threshold, first saturation information corresponding to the saturation of the detection area; When the difference from the second saturation information corresponding to the saturation of the area around the detection area is determined to be smaller than the second threshold, the first determination is made to determine that the detection area is a part corresponding to a foreign object. Procedure and
A second determination procedure for determining that the detection area is a portion corresponding to the foreign object when it is determined that the standard deviation of the brightness of the area around the detection area is smaller than a third threshold;
A program for executing a procedure for correcting the detection area when it is determined by the first determination procedure and the second determination procedure that the detection area corresponds to the foreign matter .

Images