JP6468868B2 - Image processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, an image processing method, and a program, and more particularly, to signal processing for correcting image quality degradation caused by a lens unit.

  The lens unit for coupling the subject image to the image sensor includes various factors that cause the subject image to be distorted, blurred, or deteriorated in image quality such as a decrease in peripheral light amount.

  For example, the chromatic aberration of magnification that blurs the colors around the image is that red, green, and blue light that has passed through the lens unit is imaged at different positions in the direction perpendicular to the optical axis for each wavelength. Caused by. In addition, the peripheral light amount drop in which the light amount decreases toward the end compared to the center of the image is a so-called vignetting (vignetting) or cosine fourth power law that occurs when part of the peripheral light is vignetted by the lens unit barrel. Caused by etc.

  The deterioration in image quality caused by these lens units can be reduced to some extent by combining multiple lenses with complex shapes or using fluorite with low chromatic dispersion as the lens material, for example, if chromatic aberration of magnification is used. it can. The decrease in the amount of peripheral light can also be reduced to some extent by using a lens with a large image circle. However, in any case, the image quality deterioration cannot be completely prevented.

  In view of this, a technique has been proposed in which image quality degradation due to the characteristics of the lens unit is reduced by image correction by digital signal processing.

  For example, the correction data of the peripheral light amount drop corresponding to the aperture, focal length, or shooting distance of the lens unit is recorded in the ROM, and the image is subjected to digital signal processing using the correction data corresponding to the actual state. There is a technique for correcting by this (for example, Patent Document 1).

  The image quality degradation caused by the lens unit characteristics greatly changes depending on the characteristics. Even in a certain zoom lens, the characteristics greatly change in the aperture, focal length, or photographing distance. Furthermore, in an interchangeable lens unit type imaging system, it is necessary to appropriately correct any lens unit optical system that is mounted by replacement, and it is necessary to construct a system for that purpose. Therefore, enormous costs are required to ensure accuracy and speed. Therefore, there is a technique in which not all the zoom positions, aperture values, and focus positions have a peripheral light amount fall correction value but only a discrete correction value (for example, Patent Document 2). In the case of such a configuration, correction values corresponding to the current zoom position, aperture value, and focus position are calculated and interpolated from the discrete correction values, and correction is performed only in areas where correction is required.

JP 2003-110936 A JP 2006-121384 A

  However, with the prior art disclosed in the above-mentioned patent document, it is possible to reduce the memory, but this does not guarantee the accuracy of correction and the image quality of the output image. In particular, there is a problem that the correction value must be set larger as the lens unit has a greater deterioration in optical characteristics.

  For example, when correcting an optical system in which the deterioration of the peripheral light amount is large (the peripheral light amount is large), it is necessary to prepare the system so that a larger correction can be made as the periphery of the image is approached. However, if the maximum value that can be corrected is simply set large, the correction accuracy may be sacrificed in order to maintain the circuit scale of the system. If the correction accuracy is sacrificed, an unnatural tone (tone jump) occurs in the corrected image. Further, if the peripheral light amount drop is corrected for the imaged data, an increase in noise always occurs, so it is necessary to improve the characteristics of the image sensor.

  Further, when correcting an optical system having a large amount of distortion, it is necessary to correct the amount of distortion by performing resolution conversion according to the amount of distortion. In particular, in the case of correction in which resolution conversion is performed in the enlargement direction, if the correction value is large, the deterioration in resolution in the peripheral portion becomes unnatural compared to the central portion. In order to compensate for this, it is necessary to use an advanced interpolation method or to add a sharpness separately.

  In this way, when increasing the correction value of the optical characteristics, it is not necessary to simply increase the correction value that can be set, but it is necessary to take optimum measures for the system, resulting in a complicated system and high cost. Will lead to

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a mechanism capable of dynamically calculating and correcting an optimal correction value according to the optical characteristics and the characteristics of the imaging device in a correction system that degrades the optical characteristics of the lens unit. That is.

To solve the above problems, the present invention includes information indicating the state of the optical member of the imaging device used when the image pickup means generates the image data, obtaining means for obtaining the correction data of the optical member, the acquisition Based on the state of the optical member acquired by the means and the correction data , a first correction value for correcting a decrease in image quality of the image data due to the characteristics of the optical member is set as a region of the image data or First setting means for setting for each pixel, and the correction data acquired by the acquisition means includes a predetermined flag, and the first correction value determined for each region or pixel is predetermined. In the case where a value larger than the maximum correction value is included, the first setting including the first correction value of the region or pixel where the first correction value does not reach the maximum value is included. By means By reducing the first correction value constant, if the set a second correction value for each area or pixel in the image data, which does not contain the predetermined flag on the correction data, the second A second setting unit that does not perform a process for setting the correction value , and the second correction value is used to set the image when the second correction value is set by the second setting unit. Image processing comprising: correcting means for correcting data and correcting the image data using the first correction value when the second setting value is not set by the second setting means Providing equipment.

  According to the present invention, suitable image data correction can be performed in accordance with the characteristics of the optical system of the imaging means.

It is a block diagram for demonstrating schematic structure of the imaging device which concerns on the 1st Embodiment of this invention. It is a figure showing the example of the characteristic of the peripheral light amount fall of a lens unit, and its correction value. It is a figure which shows the example of the influence of the peripheral light amount fall of a lens unit, and a correction result. It is an example of a correction value when the correction value of the peripheral light amount drop of the lens unit reaches a peak. It is a figure which shows an example at the time of image | photographing the dark to-be-photographed object with an imaging device. It is a figure which shows the influence on the noise of the dark part of FIG. It is a figure which shows the flowchart which shows the flow of a process of automatic correction ratio calculation concerning the 1st Embodiment of this invention. It is a figure which shows the flowchart for calculating the maximum correction value concerning the 1st Embodiment of this invention. It is a figure showing an example which calculates the maximum correction value of the system concerning the 1st Embodiment of the present invention. It is a figure which shows an example of the result of the calculated correction value concerning the 1st Embodiment of this invention. It is an example of the image for demonstrating the effect of the invention concerning the 1st Embodiment of this invention. It is a figure showing an example of the characteristic of distortion aberration correction of a lens unit. It is a figure which shows the example of the influence of the distortion aberration correction of a lens unit, and a correction result. It is a figure which shows the example of the correction value when the correction value of the distortion aberration correction of a lens unit reaches the peak. It is a figure which shows an example of the correction result of the distortion correction of a lens unit. It is a figure which shows the flowchart which shows the flow of the process of self-running correction ratio calculation concerning the 2nd Embodiment of this invention. It is a figure which shows an example of the file format of the lens characteristic information concerning the 2nd Embodiment of this invention. It is a figure which shows the flowchart for calculating the maximum correction value concerning the 2nd Embodiment of this invention. It is a figure which shows the method of calculating the maximum correction value concerning the 2nd Embodiment of this invention. It is a figure which shows an example of the result of the calculated correction value concerning the 2nd Embodiment of this invention. It is an example of the user interface of the application which displays the image image | photographed with the imaging device concerning the 2nd Embodiment of this invention. It is an example of the user interface of the application which displays the image by which the process of this invention concerning the 2nd Embodiment of this invention was performed.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram for explaining a schematic configuration of a photographing apparatus according to the first embodiment of the present invention. Although FIG. 1 shows the hardware configuration of the imaging apparatus, the configuration shown here is merely an example, and components other than those shown in FIG. 1 may be added. In the imaging apparatus of FIG. 1, each block may be configured by hardware using a dedicated logic circuit or a memory, except for physical devices such as an imaging element, an operation display unit, an operation unit, and a switch. Alternatively, the processing program stored in the memory may be configured by software by a computer such as a CPU executing the processing program. Hereinafter, components and functions of the imaging apparatus will be described.

  As shown in FIG. 1, the imaging apparatus 100 according to the present embodiment includes a focus lens 10a, a zoom lens 10b, an aperture shutter 12, and a lens characteristic information recording unit 10c as a lens unit 10 that is an optical member. The lens characteristic information recording unit 10c records information such as the lens ID, the focal length of the lens, and discrete correction values.

  The data recorded in the lens characteristic information recording unit 10c includes data for correcting various image quality degradation caused by the lens unit 10 such as peripheral light loss, chromatic aberration, shading, spherical aberration, and distortion by signal processing. include.

  The optical image that has passed through the focus lens 10 a and the zoom lens 10 b is incident on the image sensor 14 by opening the aperture shutter 12. The image sensor 14 photoelectrically converts the optical image and outputs it as an analog image signal. The A / D converter 16 converts the analog image signal output from the image sensor 14 into digital image data.

  The timing signal generation circuit 18 supplies a clock signal, a control signal, and the like to the image sensor 14, the A / D converter 16, and the D / A converter 26 under the control of the memory control unit 22 and the system control unit 50.

  The image processing circuit 20 performs predetermined image processing such as pixel interpolation processing and color conversion processing on the image data from the A / D converter 16 or the image data from the memory control unit 22. Further, the image processing circuit 20 performs a predetermined calculation process using the image data output from the A / D converter 16. Based on the calculation result, the system control unit 50 performs TTL autofocus processing, automatic exposure control processing, and automatic light emission control processing of the strobe 48 for controlling the aperture shutter driving unit 40 and the focus lens driving unit 42. .

  Further, the image processing circuit 20 performs predetermined calculation processing using the image data output from the A / D converter 16, and also performs TTL auto white balance processing based on the calculation result.

  The face recognition circuit 21 recognizes (detects) a human image, particularly a face, from the subject image, and notifies the system control unit 50 of the detection result.

  The system control unit 50 controls the focus lens driving unit 42 and adjusts the amount of light of the strobe 48 in order to focus on the face portion according to the face detection result in the face recognition circuit 21. Further, the system control unit 50 causes the image processing circuit 20 to perform auto white balance processing according to the face detection result in the face recognition circuit 21.

  The memory control unit 22 controls the A / D converter 16, the timing signal generation circuit 18, the image processing circuit 20, the image display memory 24, the D / A converter 26, the memory 30, and the compression / decompression circuit 32. The image data output from the A / D converter 16 is written into the image display memory 24 or the memory 30 through the image processing circuit 20 and the memory control unit 22 or only through the memory control unit 22.

  The memory 30 is used as a frame buffer for images that are continuously written on the recording medium 200 at a predetermined rate during moving image shooting. The memory 30 is also used as a work area for the system control unit 50. The compression / decompression circuit 32 reads an image stored in the memory 30, performs compression processing or decompression processing, and writes the processed data to the memory 30 again.

  The aperture shutter drive unit 40 drives the aperture shutter 12 based on the calculation result in the image processing circuit 20, and controls the aperture and shutter speed. The focus lens drive unit 42 drives the focus lens 10a based on the calculation result in the image processing circuit 20, and performs autofocus control. The zoom lens driving unit 44 drives the zoom lens 10b in accordance with a scaling operation performed by the operation unit 70. The strobe 48 has an autofocus auxiliary light projecting function and a strobe dimming function.

  The system control unit 50 includes a CPU and a memory, and controls the operation of the imaging apparatus 100 as a whole. The memory 52 stores constants, variables, computer programs (basic programs), etc. for operating the system control unit 50.

  The non-volatile memory 56 is composed of, for example, an electrically erasable and erasable EEPROM or the like, and is used as a memory for storing a computer program or the like. The computer program stored in the non-volatile memory 56 includes an application program for causing the system control unit 50 to execute processing executed by the imaging apparatus. The nonvolatile memory 56 also stores setting values set on a GUI screen such as a menu screen and an exposure correction / AEB setting screen.

  The main switch 60 is a switch for switching on / off the power supply of the imaging apparatus 100. By operating the main switch 60, the power of the recording medium 200 connected to the imaging apparatus 100 can be switched on / off at the same time.

  The shutter switch (SW1) 62 is turned on by a first stroke operation (half-pressing) of the release button, and instructs the system control unit 50 to start operations such as autofocus processing and automatic exposure control processing. The shutter switch (SW2) 64 is turned on by the second stroke operation (full depression) of the release button, and instructs the system control unit 50 to start a series of photographing processes including exposure processing, development processing, and recording processing.

  The operation unit 70 includes various buttons, switches, a touch panel, and the like. Examples of the button include a menu button, a set button, a menu movement button, and a compression mode switch. The system control unit 50 performs various operations in accordance with signals from the operation unit 70.

  The compression mode switch is a switch for selecting a compression rate for JPEG compression, or for selecting a RAW mode in which a signal from the image sensor 14 is directly digitized and recorded on the recording medium 200.

  For example, the normal mode and fine mode are available as JPEG compression modes. Select the normal mode when the data size of the captured image is important, and select the fine mode when the image quality of the captured image is important. do it.

  In the JPEG compression mode, the image data read from the image sensor 14 and written in the memory 30 is read via the A / D converter 16, the image processing circuit 20, and the memory control unit 22. The image data is compressed by the compression / decompression circuit 32 at a compression rate according to the setting, and recorded on the recording medium 200.

  In the RAW mode, image data is read out for each line in accordance with the pixel arrangement of the color filter of the Bayer arrangement of the image sensor 14 and written into the memory 30 via the A / D converter 16 and the memory control unit 22. The read image data is read and recorded on the recording medium 200. The moving image recording button 73 is operated to instruct a recording operation for recording the captured moving image data in the memory 30 or the recording medium 200.

  The I / F 90 is an interface of the recording medium 200, and the connector 92 is a connection portion with the recording medium 200. The recording medium 200 includes an I / F 204 and a recording unit 202, is detachably connected to the imaging device 100 via a connector 206, and records and saves moving image data and still image data captured by the imaging device 100. When moving image data or still image data is recorded and stored in the recording medium 200, image correction for suppressing image quality deterioration due to the characteristics of the lens unit 10 can be performed.

  The image display unit 28 sequentially displays the image signals converted into analog signals by the D / A converter 26 and written in the image display memory 24, so that an electronic viewfinder (EVF) function, so-called live view ( LV) function.

  The image display unit 28 turns on / off the electronic viewfinder (EVF) function according to an instruction from the system control unit 50. In the above live view (LV) function, image correction for suppressing image quality degradation caused by the characteristics of the lens unit 10 can be performed.

  In this image correction, when the power is turned on or when the interchangeable lens (lens unit 10) is connected, the system control unit 50 reads the characteristic information of the lens unit 10 from the lens characteristic information recording unit 10c.

  As described above, the lens characteristic information includes characteristic information such as the ID of the lens unit 10, the maximum aperture value, the focal length at the tele end, and the wide end.

  The lens state information acquisition unit 15 acquires the current state of the lens unit 10 at a predetermined timing, for example, every frame.

  Specifically, the lens state information acquisition unit 15 receives the current aperture value from the aperture shutter drive unit 40, the current focal length (zoom position) from the zoom lens drive unit 44, and the current focus from the focus lens drive unit 42. Get the position. However, it is also possible to appropriately acquire other lens states necessary for image correction.

  The system control unit 50 notifies the correction value calculation unit 23 of the lens characteristic information acquired from the lens characteristic information recording unit 10 c and the lens state information acquired from the lens state information acquisition unit 15.

  Based on the notified lens characteristic information and lens state information, the correction value calculation unit 23 performs various correction factors such as a correction coefficient for correcting a peripheral light amount drop of the lens unit 10, a correction coefficient for correcting distortion, and a correction coefficient for correcting chromatic aberration. A correction value is calculated for each region or pixel.

  The correction value here may be calculated based on a correction value calculated and recorded in advance from the lens state and lens characteristic information, or a general optical type based on the lens state information and lens characteristic information. Approximated values derived from may be used.

  The system control unit 50 recalculates the correction value calculated by the correction value calculation unit 23 using an application rate described later, and notifies the image processing circuit 20 of the recalculation. The image processing circuit 20 corrects a plurality of types of image degradation caused by the characteristics of the lens unit 10 based on the notified correction values.

  In the present embodiment, a method of correcting the peripheral light amount in the image processing circuit 20 will be described. As described above, it is preferable to apply the correction value originally calculated by the correction value calculation unit 23 to the image processing circuit 20. However, in the case of a lens unit optical system in which the decrease in the amount of peripheral light is large, correction may not be possible with the value calculated by the correction value calculation unit 23, or even if correction is possible, a favorable result may not be obtained. As an example, the maximum correction value that can be set in the image processing circuit 20 is defined, and the deterioration of the optical characteristics exceeds that value. As another example, when an image captured by the image sensor 14 and digitally output by the A / D converter 16 is corrected with the correction value calculated by the correction value calculation unit 23, an unnatural image is obtained. This is the case. This will be described with reference to FIGS.

  FIG. 2 shows how the amount of light falls when shooting with a certain lens unit. FIG. 2 is a diagram illustrating an example of the characteristics of the peripheral light loss of the lens unit and its correction value. FIG. 2A is a diagram showing how the amount of peripheral light falls under a certain photographing condition. The horizontal axis represents the image height, which is the distance from the center of the light amount, and the vertical axis represents the light amount when the light amount at the light amount center is normalized by 1. Generally, the amount of peripheral light decreases as the distance from the center of the light amount increases due to the cosine fourth power law or the vignetting of the mechanical member of the lens unit. It is assumed that the correction value calculation unit 23 possesses this value after discretization or polynomial approximation. FIG. 2B is a graph showing a correction value in a case where the correction is completely performed with respect to the light amount drop in FIG. The horizontal axis indicates the image height, which is the distance from the center of the light amount, and the vertical axis indicates the correction value (gain). In the case of this example, it is indicated that a gain close to about 10 times is required at the maximum.

  First, the case where the maximum correction value is determined in the system and the result calculated by the correction value calculation unit 23 cannot be applied as it is will be described. For example, the case where the maximum gain that can be corrected in the image processing circuit 20 is defined as five times will be described with reference to FIGS.

  FIG. 3 is a diagram illustrating an example of the influence of the peripheral light amount drop of the lens unit and a correction result, and FIG. 4 is an example of the correction value when the correction value of the peripheral light amount drop of the lens unit reaches a peak. FIG. 3A is a view showing a state in which the peripheral light quantity is not deteriorated as a subject. When FIG. 3A is photographed under the conditions of the lens unit described with reference to FIG. 2, the amount of peripheral light is reduced due to the characteristics shown in FIG. 2A, and therefore when correction is not performed, as shown in FIG. An image with dark surroundings is taken. At this time, when the maximum value that can be corrected by the image processing circuit 20 is secured with a sufficient size, an ideal image as shown in FIG. 3A can be obtained by applying the correction value shown in FIG. Can be output. However, when the maximum correction value of the image processing circuit 20 is not sufficient (when it is five times), a correction gain as shown in FIG. 4 is applied. FIG. 4 shows that the correction value is clipped at 5 times. When correction is performed using such correction values, an image as shown in FIG. 3C is output. Although the center to the intermediate image height portion of the image is corrected correctly, the correction is clipped at the periphery of the image height to generate a correction residue, and an unnatural image in which only the peripheral portion is dark is output.

  On the other hand, FIG. 5 shows an example in which an unnatural image is output when complete correction as shown in FIG. 2B is performed on an image photographed by the image sensor 14 and output by the A / D converter 16. The description will be made with reference to FIG. FIG. 5 is a diagram illustrating an example of a case where a dark subject is photographed by the imaging apparatus, and FIG. 6 is a diagram illustrating the influence of the dark portion of FIG. 5 on noise.

  An unnatural image that is output when complete correction is performed occurs as color unevenness in a peripheral portion when an image that is dark as a whole, such as a night view as shown in FIG. 5, is acquired. The principle will be described with reference to FIG. First, the line profile of the area of the peripheral portion (501) in FIG. 5 is acquired. First, as an ideal case, a case where there is no noise in the image sensor 14 will be described. At this time, the output of each color (R: red, G: green, B: blue) output by the image sensor 14 showing a linear output is the value (R: 5, G: 15, B: 5). When the subject is an achromatic color, the image processing circuit 20 applies a gain so that the above becomes an achromatic color (R: 3 times, G: 1 time, B: 3 times). In general, this processing is called white balance processing. As a result, as shown in FIG. 6B, R, G, and B have the same value (R: 15, G: 15, B: 15). At this time, when a uniform correction value (for example, 3 times) is applied to RGB in order to correct the peripheral light amount, the corrected RGB values (R: 45, G: 45, B: 45) as shown in FIG. ) Is output. Thus, when there is no noise in the image sensor 14, an image that does not cause a problem even if the peripheral light amount correction is processed without limitation is obtained.

  However, in reality, noise is generated in the image sensor 14. FIG. 6D shows a state in which noise is randomly added to FIG. Such a value is output from the actual imaging element 14 value. The amount of superimposed noise varies depending on the characteristics of the image sensor 14 and the high-sensitivity shooting mode setting (so-called ISO sensitivity) at the time of shooting. When the white balance (R: 3 times, G: 1 time, B: 3 times) as described above is applied to this output, the output is as shown in FIG. Compared with FIG. 6B, the values of R and B vary with respect to the luminance-reference G signal. When the peripheral light amount is corrected for this signal, a graph as shown in FIG. It can be seen that the difference between G, R, and B is larger than that in FIG. In particular, in the case of the achromatic subject described here, the deviation between R and B from G appears as color unevenness and is not preferable as an image. In general, the image sensor 14 often has poor characteristics (for example, linearity and color shading) at the periphery of the screen compared to the center of the screen, and the deteriorated characteristics are made conspicuous by peripheral light amount correction. There is a high possibility that In such a situation, it is preferable not to positively correct.

  In view of the above problem, the flow of the automatic correction ratio calculation processing of this embodiment will be described with reference to FIG. FIG. 7 is a flowchart showing a flow of automatic correction ratio calculation processing according to the first embodiment of the present invention. In the drawing, the step is indicated as S.

  This process is performed every time one frame of imaging is executed. If the processing speed cannot be sufficiently secured, the present process is not performed on the live view image, but the present process is performed on the still image captured in response to the shutter switch (SW2) 64 being turned on. You may do it. When the imaging operation is started, first, the focal length, the shooting distance, and the aperture position of the lens unit 10 are acquired via the lens state information acquisition unit 15. Then, correction information corresponding to the acquired focal length, shooting distance, and aperture position is acquired from the correction information stored in the lens characteristic information recording unit 10c (step 701). Next, the acquired correction information is converted into a correction value set by the image processing circuit 20 (step 702). In the case of peripheral light amount correction, this correction can be calculated by taking the reciprocal when the light amount at the center of the light amount is normalized by 1. Next, the maximum correction value Gmax applied to the actual image is calculated from the calculated correction value (step 703). In the case of an interchangeable lens unit, there are lenses that can be mounted on image pickup devices having various image pickup device 14 sizes. In that case, the maximum correction value Gmax can be obtained by calculating a correction value corresponding to the maximum image height of the image sensor.

  Detailed processing in step 703 will be described with reference to FIG. FIG. 8 is a diagram showing a flowchart for calculating the maximum correction value according to the first embodiment of the present invention. First, the system control unit 50 acquires pixel pitch information and final output image size information of the image sensor 14 from the memory 52 (step 801). Based on these two pieces of information, the maximum image height value of the output image is calculated. When the pixel pitch is P [mm] and the maximum output size is (x, y) = (a, b) [pix (number of pixels)], the maximum image height Z [mm] of the provided image is expressed by the following formula (1) ) (Step 802).

  By referring to the correction value calculated in step 702 using the maximum image height Z calculated in step 802, Gmax can be acquired.

  Next, in step 704 in FIG. 7, the maximum gain information Glim in the imaging system in which the image is taken is acquired. An example of a method for obtaining the maximum information Glim is shown in FIG. FIG. 9 is a diagram illustrating an example of calculating the maximum correction value of the system according to the first embodiment of the present invention.

  The horizontal axis indicates the ISO sensitivity at the time of shooting, and the vertical axis indicates the value of the maximum gain at each ISO sensitivity. That is, the maximum gain g2 can be applied up to ISO i1 having a relatively low sensitivity, and i2 having a relatively high sensitivity indicates that only the maximum gain g1 can be applied. As for g2, the value of the maximum gain that can be applied in a state with little noise is input, and g1 is set to a value that does not adversely affect a noisy image.

  By storing this table in the memory 52 in advance, optimal correction can be made according to the photographing conditions (ISO sensitivity). This table is preferably changed according to the performance of the image processing circuit 20 and the characteristics of the image sensor 14.

  In this way, the maximum gain Glim of the system under the photographing conditions can be acquired (step 704). Next, Gmax calculated in S703 and Glim calculated in S704 are compared (step 705). As a result, when Gmax is smaller, correction can be performed without being restricted by the system, so that correction can be performed using the correction value calculated in step 702 (step 706). On the other hand, when Gmax is larger, if correction is performed with the value as it is, an image as shown in FIG. 3C is output, and thus it is necessary to create a correction value by changing the correction ratio.

  This will be described with reference to FIG. FIG. 10 is a diagram illustrating an example of the result of the calculated correction value according to the first embodiment of the present invention. FIG. 10A is an example of a correction value when it is determined in step 705 that Gmax is larger. It is necessary to correct this correction value so as to be within the maximum gain Glim of the system, and it is necessary to avoid calculating an unnatural correction value as shown in FIG. For this purpose, it is necessary to apply a correction value as shown by a solid line (1002) in FIG. If the original correction value (1001) is a function F (x) of the image height x, the correction function G (x) (1002) to be calculated can be calculated by the following equation (2) (S707).

  By correcting with the correction value obtained from this equation (2) (step 707), it is possible to generate an unnatural image as shown in FIG. Natural corrections can be applied. In addition, since the correction value is reduced only when Gmax is larger than Glim, the conventional optimum correction is applied to the shooting conditions that do not reach Glim, compared to the method of uniformly reducing the correction ratio. can do.

(Second Embodiment)
The imaging apparatus according to the second embodiment of the present invention will be described below. In the second embodiment, distortion will be described as an example. However, portions overlapping or corresponding to those in the first embodiment will be described with reference to the drawings and symbols.

  In the case of distortion, as the amount of distortion increases, it becomes necessary to refer to a pixel far from the correction target pixel in order to correct the distortion. For this reason, the image processing circuit 20 has to process a sufficiently wide range in the work memory during the correction processing, which increases the circuit scale. Further, there is a problem that as the amount of distortion increases, the resolution of the image after distortion correction deteriorates.

  First, what kind of phenomenon will occur when work memory cannot be secured sufficiently will be described. FIG. 12 is a diagram illustrating an example of distortion correction characteristics of a lens unit under a specific condition of a certain lens unit. The horizontal axis indicates the image height, and the vertical axis indicates the amount of distortion corresponding to each image height. The value on the vertical axis represents a ratio indicating how much the image is actually distorted with respect to the ideal image formation position (in the case where there is no distortion). When the value is smaller than 1, it indicates that the image is distorted inward with respect to the ideal imaging position, and generally represents a barrel-shaped aberration. On the other hand, when the value is larger than 1, it indicates that the image is distorted outward with respect to the ideal image formation position, and generally represents a pincushion type aberration. As an example of the correction method, it is possible to correct the distortion by calculating the image height of the correction target image and replacing it with a pixel value in the case of ideal imaging according to the correction value shown in FIG. . For this reason, in real-time processing on hardware, it is necessary to perform processing by temporarily storing data in a memory in accordance with a correction value in order to retain reference destination data. If a large distortion is to be corrected, a large amount of reference area for correction must be secured in the memory each time, which results in a reduction in processing speed and an increase in cost due to an increase in memory. That is, it is necessary to determine the maximum distortion correction value in the imaging system. However, especially when the lens unit 10 is replaceable, units having various characteristics may be mounted, and some of them have a distortion amount exceeding the maximum value of the system.

  At this time, what kind of image is produced when all correction values exceeding the maximum value are clipped at the maximum value will be described with reference to FIGS. FIG. 13 is a diagram showing an example of the effect of correction of distortion aberration of the lens unit and an example of the correction result, and FIG. 14 is a diagram showing an example of correction value when the correction value of distortion aberration correction of the lens unit reaches a peak It is. FIG. 13A shows an image when a rectangular object is photographed with a lens unit having a barrel-shaped distortion. It can be seen that the higher the image height, the more the image is distorted inward, indicating the shape of the barrel. FIG. 13B is a diagram when correction is performed to the maximum. It can be seen that the distortion of the distorted rectangular object has been eliminated. FIG. 13C shows a diagram in the case where correction is performed with a clipped correction value without maximum correction due to restrictions of the imaging system. As an image of the correction value at this time, as shown in FIG. 14, the correction value is such that it is stopped at a certain value. That is, it is not preferable because the distortion remains at the tip of the peak and appears as an unnatural distortion in the image as seen in the outer peripheral portion of FIG.

  On the other hand, when the amount of distortion increases, even if the correction can be applied without being restricted by the system, the corrected image may have an unnatural result. The example will be described with reference to FIG. FIG. 15 is a diagram illustrating an example of a correction result of distortion correction of the lens unit.

  FIG. 15A is an image taken with a lens unit having barrel-shaped distortion, and it can be seen that the image is distorted toward the periphery. On the other hand, FIG. 15B is a diagram after correcting distortion aberration with respect to FIG. When comparing the size of the subjects 1501 and 1502 moving to the periphery of the screen at this time, it can be seen that the size of the subject 1502 is larger on the screen. This is equivalent to the enlargement process being performed in this region, and the resolution is degraded. On the other hand, since normal correction is not applied (or little) in the center area of the screen, there is little deterioration in resolution. That is, an unnatural image with a great difference in resolution between the center and the periphery of the screen is created. This tendency is a phenomenon that becomes more noticeable as the correction value becomes larger. From this viewpoint, it may be better to intentionally limit the correction value. As a method for compensating for the degradation of the resolution after completely correcting the distortion, a method of adjusting the frequency characteristic by interpolating the interpolation process of the enlargement process by an advanced interpolation process is conceivable. Alternatively, a method of compensating the sense of resolution by performing edge emphasis processing only on the peripheral part is conceivable, but each requires a dedicated correction circuit, leading to an increase in cost.

  As described above, similarly to the processing performed for the peripheral light amount correction in the first embodiment, it is preferable to control the correction value for distortion aberration correction according to the characteristics of the system. The basic processing flow can be applied to the same method as in FIG. Here, the description will be made with reference to FIG. 16 based on the first embodiment and further including applied processing. FIG. 16 is a diagram illustrating a flowchart showing the flow of the self-running correction ratio calculation process according to the second embodiment of the present invention. The automatic correction ratio calculation processing described in FIG. 16 can be applied not only to distortion aberration correction but also to peripheral light amount correction.

  In FIG. 16, the processing of FIG. 7 described in the first embodiment is added more efficiently, and processing that can maximize the correction effect is added. Therefore, it is necessary to have specific information in advance in the lens characteristic information recording unit 10c. First, an example of correction information possessed by the lens characteristic information recording unit 10c will be described with reference to FIG. FIG. 17 is a diagram showing an example of a file format of lens characteristic information according to the second embodiment of the present invention.

  FIG. 17 shows an example of lens unit correction information recorded in the lens characteristic information recording unit 10c. This information must be possessed for each lens unit. In the case of an imaging device having a non-exchangeable lens unit, it may be held in a memory in the imaging device, but in the case of an exchangeable lens unit, it is preferably held in the lens unit. If the lens unit is held in the lens unit, the lens unit can transmit information to the image pickup apparatus, so that there is an advantage that it is not necessary to have correction information for all the lens units on the image pickup apparatus side.

  In the memory, correction data header information 1701, which is header information for each lens unit, peripheral light amount correction data 1702, and distortion aberration correction data 1703 are recorded in respective areas. Ideally, these pieces of information should have correction data corresponding to all focal lengths, shooting distances, and aperture values, but can be reduced in accordance with the capacity of the memory and the characteristics of the correction information. . Here, a correction value is recorded in accordance with the optical characteristics of the lens unit, and if correction is made using the correction value described in this area, a value that completely corrects the peripheral light amount and distortion is recorded. Is desirable.

  Next, details of the correction data header information 1701 will be described. The correction data header information 1701 possesses lens ID information 1704 that is lens unit identification information, peripheral light amount correction reference data 1705, and distortion aberration correction reference data 1706. The peripheral light amount correction reference data 1705 includes a peripheral light amount data size 1707 and a peripheral light amount data offset 1708 indicating an offset amount to the head address of the correction data. The peripheral light amount correction reference data 1705 includes a peripheral light amount reduction determination flag 1709 that is a determination flag indicating whether or not reduction processing is necessary, and peripheral light amount reduction determination image height information 1710 used for determination of reduction processing described later. The distortion aberration correction reference data 1706 includes a distortion aberration data size 17011 and a distortion aberration data offset 1712 indicating an offset amount to the start address of the correction data. The distortion aberration correction reference data 1706 includes a distortion aberration reduction determination flag 1713 that is a determination flag for determining whether reduction processing is necessary, and distortion aberration reduction determination image height information 1714 that is used for determination of reduction processing described later.

  The processing when the lens characteristic information recording unit 10c has the correction data having the above-described file configuration will be described with reference to FIG. Since the basic configuration is the same as that described with reference to FIG. 7 in the first embodiment, differences will be mainly described.

  First, the process of acquiring the shooting conditions (step 1601) and acquiring the correction value (step 1602) may be the same process as step 701 and step 702, respectively. Next, a reduction process determination flag (in the case of distortion aberration correction, distortion aberration reduction determination flag 1713) recorded in the lens characteristic information recording unit 10c is referred to. If the flag is not set, correction is performed using the correction value calculated in step 1602 without performing reduction processing (step 1603). Thereby, an unnecessary calculation can be avoided. In step 1605, a maximum correction value Gmax for a predetermined image height is calculated.

  A method for calculating the maximum correction value Gmax for the predetermined image height will be described with reference to FIG. FIG. 18 is a diagram illustrating a flowchart for calculating the maximum correction value according to the second embodiment of the present invention. First, similarly to step 801, the pixel pitch information and final output size of the image sensor are acquired (step 1801). Next, determination image height information (in the case of distortion aberration correction, distortion aberration reduction determination image height information 1714) recorded in the lens characteristic information recording unit 10c is acquired (step 1802). The value to be recorded here is a value for inputting in advance which image height is not clipped when the correction value is clipped at the time of shooting. Next, a maximum correction value is calculated in step 1803 based on the information acquired in step 1802.

  The process of calculating the maximum correction value in step 1803 will be described with reference to FIG. FIG. 19 is a diagram illustrating a method for calculating the maximum correction value according to the second embodiment of the present invention. A solid line 1901 indicates the correction value calculated in step 1602. Let D be the maximum image height calculated in step 1801. Further, if the determination image height information acquired in step 1802 is α, Gmax can be obtained as a distortion rate at the image height αD.

  Next, as in step 704, the maximum correction value Glim of the system is calculated (step 1606). In the first embodiment, it has been described that the maximum correction value of the system is changed for each ISO sensitivity with reference to FIG. 9, but in the case of distortion aberration correction, since the change in image quality is small depending on the ISO sensitivity, it is a uniform value. You may keep it.

  Next, Gmax calculated in step 1605 is compared with Glim calculated in step 1606 (step 1607). If Gmax has a smaller correction value, it is not necessary to perform a reduction process, and correction is performed using the correction value calculated in step 1602 (step 1604). On the other hand, when the correction value is large, a correction ratio R (step 1608) is calculated so that the determination image height falls within Glim. The calculation method of R may be the same as the equation (2). By correcting with the correction ratio calculated in this way (step 1609), a correction effect can be obtained as compared with the case of determining the ratio with the maximum image height as in the first embodiment. This will be described with reference to FIG.

  FIG. 20 is a diagram illustrating an example of the result of the calculated correction value according to the second embodiment of the present invention. A dotted line 2001 represents a correction value calculated from step 1602 when the aberration is completely corrected, and is a condition when the maximum correction value Glim of the system is exceeded. On the other hand, a dotted line 2002 is a correction value when the correction ratio is reduced so that the correction value at the maximum image height does not exceed Glim as shown in the first embodiment. It can be seen that the correction value of Glim is obtained at the maximum image height D. A solid line 2003 is the result of processing by the algorithm of this embodiment. The maximum correction value Glim of the system is reached at a predetermined image height αD. However, it can be seen from the dotted line 2002 that the correction value shows the correction effect. Here, if α is set too small, it becomes as shown in FIG. 3C or FIG. 13C. Therefore, when the value of α is input to the lens characteristic information recording section, care should be taken. There is a need.

  Finally, a case will be described in which an image captured in the first embodiment or the second embodiment is reprocessed by post-processing. When photographing is performed by the method of the first embodiment or the second embodiment, there may be a case where sufficient correction is not applied due to restrictions of the imaging device. However, there are cases where some users want to sufficiently perform lens unit optical deterioration correction processing even if there is some image corruption.

  Such a user can record an image before image processing (the above-mentioned DRAW image) and correct it using an external application such as a PC after shooting. An example of the user interface of the application is shown in FIG.

  In FIG. 21, an image display area 2101 on the application is an area for displaying images in the folder selected by the user. The optical correction application selection area 2102 is an area for performing display for enabling the user to set for each correction process whether or not to apply optical correction to the displayed image. Specifically, checking the horizontal check box labeled “Apply” means that the corresponding optical correction has been applied. Further, for an image selected by the user, it is possible to newly record an image processed under the conditions set by pressing the OK button 2103 after the processing on the application is completed.

  In particular, the above-described RAW image is converted into a general image file (JPEG or BMP) by post-processing by a user using a PC or the like after shooting. When this RAW image is processed by an application, it is desirable to display the image that is first displayed before the user edits the image after converting it to the setting at the time of shooting of the camera body. In other words, when the setting is such that optical correction is applied when a RAW image is captured, it is expected that the optical correction is applied to an image to be displayed first. However, since the RAW image is data before image processing, it is necessary to apply optical correction when displaying the image on the application side.

  Here, in the case of shooting, if the desired correction can be performed without system restrictions, an interface as shown in FIG. 21 (ON / OFF by an application check box) may be used. However, when the system is limited as in the present invention and sufficient correction cannot be applied at the time of shooting, the interface as shown in FIG. 21 cannot meet the expectation of the user who wants to apply sufficient correction.

  In general, an application that operates on a PC is not easily restricted by an imaging device. For this reason, the interface as shown in FIG. 21 has a problem that there is no means to execute it, although sufficient correction can be realized. Further, since it is difficult for the user to know which image is applied with the algorithm of the present embodiment and the correction effect is reduced, it is preferable to provide a user interface that allows the user to check the corrected result by reducing the correction effect.

  Therefore, when a determination flag indicating whether or not the algorithm of the present invention has been applied is recorded in the image file, and the value is read by the application side and described as being applied, the interface shown in FIG. It is preferable to change to. Basically, the optical correction application selection area 2201 is changed. When the user selects an image shot with the correction ratio reduced according to the present invention, the image is displayed in a format such as the slider 2202 or the slider 2203 instead of the check box. When a captured image is opened for the first time, the correction ratio corrected at the time of shooting is recorded in a file and the value is displayed. For example, assuming that the value of Gmax is 100 and corresponds to the right end of the slider, the value of Glim with respect to Gmax is expressed by a numerical value between 0 and 100. For example, when Gmax is a gain of 10 times, if Glim is a gain of 5 times, the slider may be displayed at the middle 50 position. By doing in this way, the user can determine whether or not the reduction process has been applied at the time of shooting, and the user who wants to perform further correction adjusts the slider while confirming the image of the application result, A desired correction can be applied.

  By taking the above measures, it is possible to apply the optimum correction in the combination of the imaging system and the lens unit, and the user can arbitrarily change the correction value by post-processing.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 10 Lens unit 10c Lens characteristic information recording part 15 Lens state information acquisition part 20 Image processing circuit 23 Correction value calculation part 50 System control part

Claims (11)

Information indicating the state of the optical member used when the imaging means generates image data, and acquisition means for acquiring correction data of the optical member ;
Based on the state of the optical member acquired by the acquisition unit and the correction data , a first correction value for correcting a decrease in image quality of the image data due to the characteristics of the optical member is set as the first correction value of the image data. First setting means for setting for each region or pixel;
The correction data acquired by the acquisition means includes a predetermined flag, and the first correction value obtained for each area or pixel has a value larger than a predetermined maximum correction value. If it is included, the first correction value set by the first setting means is reduced including the first correction value of the region or pixel where the first correction value does not reach the maximum value. Then, a second correction value is set for each area or pixel of the image data, and when the predetermined flag is not included in the correction data, a process for setting the second correction value is performed. A second setting means not to perform;
If the second correction value is set by the second setting means, the second of said image data corrected using the correction value, the second correction value by said second setting means An image processing apparatus comprising: a correction unit that corrects the image data using the first correction value when it is not set .   The image processing apparatus according to claim 1, wherein the deterioration in image quality due to the characteristics of the optical member is at least one of a decrease in image quality due to a decrease in peripheral light amount and a decrease in image quality due to distortion. The correction data includes first correction data for correcting a decrease in image quality due to a decrease in peripheral light amount, and second correction data for correcting a decrease in image quality due to distortion.
The said 2nd setting means determines whether the said predetermined | prescribed flag is contained with respect to each of the said 1st correction data and the said 2nd correction data, The said 2nd setting means is characterized by the above-mentioned. Image processing device. The state of the optical members, the zoom position, focus position, and the image processing apparatus according to any one of claims 1 to 3, characterized in that it comprises at least one aperture. The second setting unit is configured such that the first correction value at the maximum image height of the image data output from the imaging unit or the first correction value at a preset image height is the predetermined correction value. The first correction value set by the first setting means is reduced, and a second correction value is set for each area or pixel of the image data when The image processing apparatus according to any one of claims 1 to 4 . The maximum value of the correction value, said predetermined, the image processing apparatus has a memory size, and, according to claim 1, wherein the set based on at least one of the image quality of the image data The image processing apparatus according to any one of the above. The image according to any one of claims 1 to 6 , wherein the second setting unit sets the second correction value so as not to exceed a predetermined maximum correction value. Processing equipment. A slider indicating the maximum value of the first correction value obtained for each of the regions or pixels and the maximum value of the predetermined correction value when the second setting means sets the second correction value; and The image processing apparatus according to claim 1, further comprising a control unit that causes the display device to display at least one of the numerical values. The imaging means;
Imaging apparatus characterized by having an image processing apparatus according to any one of claims 1 to 7. Information indicating the state of the optical member used when the imaging means generates image data, and an acquisition step of acquiring correction data of the optical member ;
Based on the state of the optical member acquired in the acquisition step and the correction data , a first correction value for correcting a decrease in image quality of the image data due to the characteristics of the optical member is obtained. A first setting step for setting for each region or pixel;
The correction data acquired in the acquisition step includes a predetermined flag, and the first correction value obtained for each area or pixel has a value larger than a predetermined maximum correction value. If it is included, the first correction value set in the first setting step is reduced including the first correction value of the region or pixel where the first correction value does not reach the maximum value. Then, a second correction value is set for each area or pixel of the image data, and when the predetermined flag is not included in the correction data, a process for setting the second correction value is performed. A second setting step not to be performed ;
If the second correction value is set in the second setting step, the second said image data corrected using the correction value, the second correction value in the second setting step is An image processing method comprising: a correction step of correcting the image data using the first correction value when not set . The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 8 .