JP7744799B2 - Imaging device and control method thereof, lens device, image magnification correction data, image processing device and image processing method, and accessory device - Google Patents

Description

The present invention relates to an imaging device and a control method thereof, a lens device, image magnification correction data, an image processing device and image processing method, and an accessory device.

Technology for correcting changes in the angle of view (breathing) caused by movement of the focus lens is known (Patent Document 1).

Japanese Patent Application Laid-Open No. 2008-42405

In Patent Document 1, breathing is corrected by cropping an image captured at a focus lens position on the close-up side, where the angle of view is wider, with infinity, which provides the narrowest angle of view, as the reference point. As a result, unnecessary cropping may occur on images captured with a limited focus range (the range of movement of the focus lens).

Furthermore, when correcting for breathing, which results in the narrowest angle of view at the closest end, images captured with the focus lens positioned on the infinity side are enlarged and cropped. Even in this case, unnecessary enlargement and cropping may occur on images captured with the focus range (drive range of the focus lens) limited.

The present invention was made in consideration of these problems with the prior art. In one aspect, the present invention provides an imaging device and a control method thereof, a lens device, image magnification correction data, an image processing device and image processing method, and an accessory device for appropriately correcting breathing in images captured with a limited focus range.

The above-mentioned object can be achieved by an imaging device having, for example, an acquisition means for acquiring, from a lens device, information regarding image magnification correction according to the state of a component that sets the focus range of the lens device, and a correction means for correcting the angle of view of an image captured using the lens device based on the information regarding image magnification correction and information regarding the state of the lens device at the time of capturing the image, wherein the information regarding the state of the lens device at the time of capturing the image includes the state of the component.

The present invention provides an imaging device and its control method, a lens device, image magnification correction data, an image processing device and image processing method, and an accessory device for appropriately correcting breathing in images captured with a limited focus range.

1 is a schematic block diagram of a lens device and an imaging device according to the present invention; FIG. 1 is a schematic diagram showing the relationship between the position of the focus lens and the correction magnification of the present invention. FIG. 10 is a schematic diagram showing the correction magnification in a limited focus driving range state according to the present invention; FIG. 1 is a schematic diagram showing the relationship between the position of the focus lens and the correction magnification of the present invention. 1 is an operation flow diagram of the image magnification correction process of the present invention; A lens communication flow diagram between the lens device of the present invention and the imaging device. FIG. 1 is an explanatory diagram of the structure of image magnification correction data according to the present invention; White balance control flow chart

The present invention will now be described in detail based on illustrative embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although the embodiments describe multiple features, not all of them are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the accompanying drawings, the same reference numbers are used to designate identical or similar components, and redundant explanations will be omitted.

The following description will focus on an embodiment of the present invention in which it is implemented using a digital camera. However, imaging functions such as an image sensor or imaging optical system are not essential for the present invention. Therefore, the present invention can be implemented in any electronic device capable of handling image data. In addition to imaging devices, such electronic devices include computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

● (First embodiment)
1 is a block diagram showing an example of the functional configuration of an imaging system including a lens device and an imaging device according to an embodiment of the present invention. A lens device 150 is detachably attached to the imaging device 100 via a lens mount unit 180. The lens device 150 may be integrated into the imaging device 100.

First, the internal configuration of the imaging device 100 will be described. The imaging element 102 is, for example, a CCD image sensor or a CMOS image sensor. The imaging element 102 has a plurality of pixels, each having a photoelectric conversion unit, arranged two-dimensionally, and converts the optical image generated by the lens device 150 into a group of pixel signals (analog image signals). The imaging element 102 has, for example, a primary color Bayer array color filter. Therefore, each pixel has a color filter of one of R (red), G (green), or blue (B). Note that color filters other than the primary color Bayer array may also be used.

For example, the image sensor 102 may use a color filter with an array of complementary color filters. Furthermore, the image sensor 102 is movable in the x-axis and y-axis directions of a plane perpendicular to the optical axis of the lens device 150 to suppress image blur. The movement of the image sensor 102 is controlled by the system control unit 130 via the imager drive unit 109.

The image generation unit 103 converts the analog image signal obtained from the image sensor 102 into a digital image signal. The digital image signal is input to the memory control unit 105 and the image processing unit 140.

The timing generation unit 104 supplies clock signals and synchronization signals to the image sensor 102, image generation unit 103, memory control unit 105, system control unit 130, and image processing unit 140.

The memory control unit 105 controls the image generation unit 103, timing generation unit 104, image display device 106, memory 107, recording unit 108, and image processing unit 140. The digital image signal from the image generation unit 103 is written to the memory 107 and recording unit 108 via the image processing unit 140 and memory control unit 105.

The image display device 106 is, for example, an LCD or organic EL display. When the imaging device 100 is in shooting mode, the image display device 106 sequentially displays moving images captured using the image sensor 102, thereby realizing an electronic viewfinder (EVF) function. When the imaging device 100 is in playback mode, the image display device 106 displays images recorded in the memory 107 and recording unit 108. The image display device 106 also displays user interfaces such as menu screens, information related to the imaging system, and the like.

The memory 107 stores still image data and video data obtained by shooting or read from the recording unit 108, and is used as a work area for the system control unit 130. A portion of the memory 107 may be used as video memory for the image display device 106. The memory 107 also has a non-volatile area that stores settings for the imaging device 100, programs executed by the system control unit 130, GUI data, and the like. The non-volatile area of the memory 107 also stores data related to optical aberration correction and image magnification correction for the lens device 150, which will be described later.

The recording unit 108 is, for example, a removable memory card. Images obtained by shooting are stored in a data file and recorded in the recording unit 108. The recording unit 108 may also be a storage device built into the imaging device 100.

The shutter control unit 110 drives the shutter 101 based on the control of the system control unit 130.

SW1 (switch 1) 115 is turned on by half-pressing the shutter switch. The system control unit 130 recognizes the turning on of SW1 as an instruction to prepare for still image capture, and executes predetermined preparatory operations for capture. These preparatory operations may include AF (autofocus) processing, AE (auto exposure control) processing, AWB (automatic white balance adjustment) processing, etc.

SW2 (switch 2) 116 is turned on when the shutter switch is fully pressed. The system control unit 130 recognizes the turning on of SW2 as an instruction to start capturing a still image, and executes a predetermined still image capturing operation. The still image capturing operation may include exposing the image sensor 102 in accordance with exposure conditions, generating a still image data file for recording from an image signal obtained from the image sensor 102, and recording the generated still image data file in the recording unit 108.

The camera operation unit 117 is a general term for input devices such as switches, buttons, and keys that are provided so that the user can give instructions to the imaging device 100. The input devices that make up the camera operation unit 117 have names that correspond to the functions assigned to them. For example, the camera operation unit 117 includes a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, an enter key, and the like.

The functions assigned to the same input device may be variable. The input device may also be software buttons or keys using a touch display. The operation unit 117 may also include an input device that supports non-contact input methods, such as voice input or eye-gaze input.

The gyro sensor 118 generates a signal corresponding to the attitude of the image capture device 100 and outputs it to the system control unit 130. The signal output by the gyro sensor 118 is used to control the movement of the image capture element 102 for image blur correction and the drive of the shift lens 154.

The I/F (interface) 120 is a communication interface with the lens device 150. The I/F 120 is electrically connected to the I/F 170 of the lens device 150 via the connector 190. The system control unit 130 can communicate with the lens control unit 160 via the I/F 120. The system control unit 130 acquires information about the lens device 150 and controls the operation of the lens device 150 by communicating with the lens control unit 160 via the I/F 120.

The connector 190 is an electrical contact point that transmits signals and power between the imaging device 100 and the lens device 150.

The system control unit 130 is one or more processors (CPUs) capable of executing programs. By executing programs stored in the non-volatile area of the memory 107, the system control unit 130 controls the operation of the imaging device 100 and lens device 150 and realizes the functions of the imaging system. The system control unit 130 monitors the operation of the shutter switches 115, 116 and the camera operation unit 117, and when an operation is detected, it performs processing according to the operation.

The AF control unit 131, AE control unit 132, lens communication control unit 133, and optical correction control unit 134 within the control unit 130 are functional blocks that represent some of the operations performed by the system control unit 130. Therefore, the operations of these control units 131 to 134 are actually realized by the system control unit 130 executing a program.

The AF control unit 131 is responsible for the AF processing of the imaging device 100. The AF control unit 131 determines the focus lens drive amount based on information such as the focal length and the position of the focus lens 151 obtained from the lens device 150 via the I/F 120, and an AF evaluation value generated by, for example, the image processing unit 140. The AF control unit 131 controls the position of the focus lens 151 by sending a control signal including the focus lens drive amount to the lens device 150 via the lens communication control unit 133 and the I/F 120.

The AE control unit 132 is responsible for the AE processing of the image capture device 100. The AE control unit 132 determines exposure conditions (aperture value, shutter speed, ISO sensitivity, etc.) based on information such as the maximum aperture F-number and focal length obtained from the lens device 150 via the I/F 120, and on AE evaluation values generated by the image processing unit 140, for example. The AE control unit 132 controls the opening size of the diaphragm 153 by sending a control signal including the aperture value to the lens device 150 via the lens communication control unit 133 and the I/F 120. The AE control unit 132 also controls the gain used by the image capture element 102 based on the ISO sensitivity.

The lens communication control unit 133 is responsible for communication processing between the imaging device 100 and the lens device 150. When the attachment of the lens device 150 is detected via the I/F 120, the imaging device 100 becomes able to communicate with the lens device 150 via the lens communication control unit 133.

For example, when the system control unit 130 transmits an information request signal, the lens control unit 160 transmits information about the lens device 150 to the imaging device 100 by performing operations such as those shown in the flowchart of FIG. 5, which will be described later. The imaging device 100 and lens device 150 can communicate at any timing or at timing based on an imaging synchronization signal output from the timing generation unit 104. When an imaging synchronization signal is input from the timing generation unit 104, the lens device 150 may transmit status information, which is information about the current status of the lens device 150, in synchronization signal communication mode. The status information may include focus lens position, focus lens state, aperture value, focal length, and variable aberration amount. Note that the variable aberration amount is transmitted by specific models of lens devices 150 and may not be included.

The optical correction control unit 134 reads information related to optical aberration correction, including design data, and data related to image magnification correction from the memory 107 in response to information received from the lens device 150, and determines a correction value based on the read information according to the status information of the lens device. The optical correction control unit 134 sets the determined correction value in the image processing unit 140, thereby controlling processing related to optical aberration correction in the image processing unit 140, such as image magnification correction, distortion aberration correction, chromatic aberration of magnification correction, peripheral illumination correction, and image restoration.

The image processing unit 140 applies various processes to digital image signals to generate signals and image data, and acquire and/or generate various types of information. The image processing unit 140 may be a dedicated hardware circuit, such as an ASIC, designed to achieve specific functions, or may be configured so that at least some of the functions are achieved by a processor, such as a DSP, executing a program.

The image processing applied by the image processing unit 140 includes color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like.
Color interpolation is a process of interpolating values of color components that cannot be obtained during shooting, and is also called demosaic processing.
The correction processing includes white balance adjustment, gradation correction, correction of image degradation caused by optical aberrations in the imaging optical system of the lens device 150 (image restoration), correction of the effects of peripheral light falloff in the imaging optical system, color correction, and the like.
The detection process includes detection of characteristic regions (for example, face regions or human body regions) and their movements, and person recognition processing.
The data processing includes processes such as synthesis, scaling, encoding and decoding, and header information generation (data file generation).
The evaluation value calculation process includes processes such as generating signals and evaluation values used in autofocus (AF) detection, and generating evaluation values used in automatic exposure (AE) control.
Special effect processing includes adding a blur effect, changing color tones, relighting, and the like.
The above-described processes are examples of processes that can be applied by the image processing unit 140, and do not limit the processes that can be applied by the image processing unit 140.

The image processing unit 140 can generate image data according to the intended use. Of this data, image data for recording is stored in a data file in a format according to the settings and recorded in the recording unit 108. Furthermore, image data for display is displayed on the image display device 106.

The resizer 141, image magnification correction unit 142, distortion aberration correction unit 143, magnification chromatic aberration correction unit 144, peripheral illumination correction unit 145, and image restoration unit 146 within the image processing unit 140 are functional blocks that represent some of the operations performed by the system control unit 130. Therefore, the operations of these functional blocks 141 to 146 are actually performed by the image processing unit 140.

The image magnification correction unit 142 corrects the breathing of the lens device 150 to the reference angle of view by applying enlargement and cropping processes to the image with a wider angle of view. The operation of the image magnification correction unit 142 will be described in detail later.

The distortion correction unit 143 applies distortion correction processing to the digital image signal based on the distortion correction data acquired by the optical correction control unit 134.

The magnification chromatic aberration correction unit 144 applies magnification chromatic aberration correction processing to the digital image signal based on the magnification chromatic aberration correction data acquired by the optical correction control unit 134.

The peripheral illumination correction unit 145 applies peripheral illumination correction processing to the digital image signal based on the peripheral illumination correction data acquired by the optical correction control unit 134.

The image restoration unit 146 applies image restoration processing to the digital image signal based on the correction data for image restoration acquired by the optical correction control unit 134.

Next, we will explain the lens device 150. The lens device 150 has an imaging optical system composed of multiple optical components, including a focus lens 151, a zoom lens 152, and a shift lens 154, which are movable lenses.

The focus lens 151 moves in the optical axis direction to adjust the focal distance of the imaging optical system.
The zoom lens 152 moves in the optical axis direction to adjust the focal length (angle of view) of the imaging optical system.
The aperture 153 has a variable aperture diameter (aperture value) and adjusts the amount of light incident on the imaging device 100 from the lens device 150 .
The shift lens 154 is movable in the x-axis direction and the y-axis direction, which are perpendicular to the optical axis direction and perpendicular to each other, and the movement is controlled so as to suppress image blur caused by the movement of the imaging device 100 .

The focus control unit 155 drives the focus lens 151 under the control of the lens control unit 160 or the lens operation unit 161. The focus control unit 155 also outputs focus information such as the current position of the focus lens 151 or the focal length of the imaging optical system to the lens control unit 160.

The zoom control unit 156 drives the zoom lens 152 under the control of the lens control unit 160 or the lens operation unit 161. The zoom control unit 156 also outputs zoom information such as the current focal length of the imaging optical system to the lens control unit 160.

The zoom lens 152 may have an aberration-variable lens for varying the amount of optical aberration in the imaging optical system. The movement of the aberration-variable lens in the optical axis direction is also controlled by the zoom control unit 156. If the zoom lens 152 has an aberration-variable lens, the zoom control unit 156 outputs information regarding the amount of variable aberration to the lens control unit 160.

The aperture control unit 157 drives the aperture 153 under the control of the lens control unit 160 or the lens operation unit 161, and controls the opening size (aperture value) of the aperture 153. The aperture control unit 157 also outputs aperture information such as the current aperture value to the lens control unit 160.

The angular velocity detection unit 158 detects the angular velocity of the lens device 150 in a specific direction (e.g., the Yaw and Pitch directions) and outputs it to the lens control unit 160.

Under the control of the lens control unit 160, the optical image stabilization control unit 159 drives the shift lens 154 based on angular velocity information of the lens device 150 and information based on the output of the gyro sensor 118 of the imaging device 100. The optical image stabilization control unit 159 also outputs position information of the shift lens 154 to the lens control unit 160.

The lens information storage unit 165 is a non-volatile memory that stores static information about the lens device 150, such as design data related to the optical aberration of the lens device 150 (imaging optical system), design data related to image magnification correction, and the presence or absence and number of states of a focus limit switch. The lens information storage unit 165 can be referenced by the lens control unit 160.

The lens control unit 160 includes, for example, a processor (CPU) capable of executing a program, a non-volatile memory for storing the program, and a memory for loading the program to be executed. The lens control unit 160 controls the operation of each functional block by executing the program with the CPU. The lens control unit 160 also transmits information about the lens device 150 in response to a request from the system control unit 130, and controls the driving of movable members such as the aperture 135 and the focus lens 151.

The lens operation unit 161 is a general term for input devices that are provided on the housing of the lens device 150 and allow the user to switch and adjust settings related to the operation of the lens device 150. The lens operation unit 161 may include, for example, a focus limit switch, a focus operation ring, a zoom operation ring, an AF/MF switch, an anti-vibration on/off switch, an aberration variable control ring, etc. These are examples, and other operation members may also be included.

The focus limit switch is a switch that specifies the focus range (focusing range or driving range of the focus lens 151) of the lens device 150. In this embodiment, as an example, the lens device 150 has a focus limit switch that can specify the following three focus ranges (modes):
Full mode: This mode does not limit the driving range of the focus lens 151. The focus lens is driven over the entire movable range (for example, the range from the closest end (<0.3 m) to the infinity focusing distance).
Macro Mode This mode limits the focusing distance to a close range (for example, 0.3 m to 0.5 m) suitable for macro photography, and the focus lens 151 is driven.
Normal Mode This mode drives the focus lens 151 within a range of focusing distances (for example, 0.5 m to infinity) suitable for general photography such as portrait photography and distant views.
Note that modes other than the full mode are merely examples, and the number and types of modes are arbitrary.

For lens devices 150 that do not have a focus limit switch, the imaging device 100 can be treated as a lens device that only has a full mode or a normal mode. Furthermore, the focus range of the lens device 150 can also be controlled by the settings of the imaging device 100 in the same way as the focus limit switch. Therefore, the system control unit 130 recognizes the focus range by taking into account the presence and/or state of the focus limit switch, as well as the presence and/or state of the focus range settings of the imaging device 100.

When the lens control unit 160 detects an operation of the lens operation unit 161, it transmits the operation details to the imaging device 100 (system control unit 130) via the I/F 170. When the system control unit 130 receives the operation details of the lens operation unit 161 via the I/F 120, it switches the operating mode of the functions of the lens device 150 according to the operation details.

The I/F (interface) 170 is a communication interface with the imaging device 100. The I/F 170 is electrically connected to the I/F 120 of the imaging device 100 via the connector 190. The lens control unit 160 can communicate with the system control unit 130 via the I/F 170. The lens control unit 160 transmits information about the lens device 150 and controls the operation of the lens device 150 by communicating with the system control unit 130 via the I/F 170.

Next, the image magnification correction process executed by the imaging device 100 in this embodiment will be described.
The image magnification correction process corrects the angle of view variation (breathing) of the lens device 150 (image pickup optical system) that occurs when the focus lens 151 is driven. Here, the angle of view variation is corrected by applying enlargement and cropping processes to an image captured at an angle of view wider than the reference angle of view to generate an image equivalent to the reference angle of view. In the present invention, the system control unit 130 can control the image magnification correction process by setting an angle of view correction value for correcting the angle of view variation in the image magnification correction unit 142 of the image processing unit 140.

Next, we will explain how to determine the angle of view correction value. The system control unit 130 can determine the angle of view correction value by referring to information related to image magnification correction, which will be described later, information related to the lens device 150, and information related to the imaging device 100.

The information related to image magnification correction is information for correcting variations in the angle of view according to the position of the focus lens (the focal length of the lens device 150), and includes the correction magnification, reference magnification, and maximum magnification. These magnifications change according to the focal length, focal length, variable aberration amount, and the state of the focus limit switch of the lens device 150. Furthermore, the amount of information included in the information related to image magnification correction may vary depending on the model of the lens device 150. Therefore, information related to image magnification correction suitable for the lens device 150 is pre-stored in the lens information storage unit 165 of the lens device 150.

Instead of the correction magnification, other information from which the correction magnification can be obtained may be used. For example, the angle of view fluctuation rate according to the position of the focus lens can be used. The correction magnification can be obtained by obtaining the reciprocal of the angle of view fluctuation rate.

The system control unit 130 acquires information related to image magnification correction from the lens device 150 through communication with the lens control unit 160 and stores it in memory 107. Note that information related to image magnification correction may also be acquired from an external device with which communication is possible via an external interface (such as a USB, wireless LAN, or Bluetooth (registered trademark) interface) possessed by the imaging device 100. Alternatively, information related to image magnification correction for multiple lens device models may be stored in memory 107 in advance, and information corresponding to the model of the attached lens device 150 may be acquired from memory 107.

The system control unit 130 obtains information about the current state of the lens device 150, such as the focus lens position, focal length, and state of the focus limit switch, from the lens device 150 at any timing and/or at timing synchronized with the imaging frame.

The system control unit 130 also acquires information about the image capture device 100, such as shooting conditions based on AE processing and user settings. The system control unit 130 then determines the angle of view correction value using the acquired information about the lens device 150 and the image capture device 100.

The information related to image magnification correction will be explained in detail. Figure 2 is a diagram that schematically shows an example of the relationship between the focus lens position of a lens device and the correction magnification. For ease of understanding, in Figure 2 and the following explanation, the focal distance corresponding to the focus lens position may be used instead of the focus lens position.

In the example shown by solid line 201 in Figure 2, the correction magnification is 100% when the focus lens is positioned at a focal distance of 0.3 m. A magnification of 100% corresponds to 1x, meaning that the image remains at its original size (not enlarged). Similarly, the correction magnifications when the focus lens is positioned at a focal distance of 0.5 m and infinity are 130% and 150%, respectively. In this way, a lens device with the relationship shown in Figure 2 has the characteristic that the angle of view on the infinity side becomes wider due to breathing.

By setting the correction magnification in the image magnification correction unit 142 as the angle of view correction value according to the relationship of the solid line 201, the system control unit 130 can correct changes in the angle of view caused by changes in the focus lens position (focal distance). This is the image magnification correction process.

The relationship shown in Figure 2 changes depending on the state of the lens device 150 (focal length and variable aberration). Therefore, relationships are stored for multiple states of the lens device 150. If the relationship in one state can be converted to the relationship in another state, the reference relationship and the information required for the conversion may be stored. If the relationship can be expressed as a function of the state, the function may be stored. When storing relationships for each state, multiple discrete states can be stored to reduce the amount of storage required, as described below.

Figure 3 is a diagram that schematically illustrates an example of the relationship between the focus lens position and correction magnification depending on the state of a focus limit switch when the lens operation unit 161 of the lens device 150 has the focus limit switch. Here, as explained above, it is assumed that the focus limit switch can be set to either full mode, macro mode, or normal mode.

Figure 3(a) is a diagram showing a schematic example of the relationship between correction magnification and focus lens position when the focus limit switch is in full mode. 302 indicates that the correction magnification at the focus lens position corresponding to a focusing distance of 0.3 m from the closest end is the reference magnification. 303 indicates that the correction magnification is maximum at the focus lens position corresponding to a focusing distance of infinity, which is 150% when the reference magnification is 100%. When the focus limit switch is in full mode, image magnification correction is possible for focusing distances from the closest end to infinity (the entire movable range of the focus lens), but the image is enlarged by a maximum of 150% (1.5x).

Figure 3(b) is a schematic diagram showing an example of the relationship between correction magnification and focus lens position when the focus limit switch is in macro mode. In macro mode, the focus lens is driven within a limited focal distance range of 0.3 to 0.5 m. 305 indicates that the correction magnification at the focus lens position corresponding to a focal distance of 0.3 m is the reference magnification. 306 indicates that the correction magnification is maximum at the focus lens position corresponding to a focal distance of 0.5 m, which is 130% when the reference magnification is 100%. When the focus limit switch is in macro mode, the maximum magnification is 130%, allowing the maximum magnification relative to the reference magnification to be reduced more than in full mode.

Figure 3(c) is a schematic diagram showing an example of the relationship between correction magnification and focus lens position when the focus limit switch is in normal mode. In normal mode, the focus lens is driven within a limited range of focal distances from 0.5 m to infinity. 308 indicates that the correction magnification at the focus lens position corresponding to a focal distance of 0.5 m is the reference magnification. 309 indicates that the correction magnification is maximum at the focus lens position corresponding to a focal distance of infinity, which is 115% when the reference magnification is 100%. When the focus limit switch is in normal mode, the maximum magnification is 115%, allowing the maximum magnification to be reduced relative to the reference magnification compared to full mode and macro mode.

In this way, in this embodiment, by changing the correction magnification that serves as the reference magnification according to the drive range of the focus lens, it is possible to suppress the maximum magnification of the image within the drive range of the focus lens. Since image quality deteriorates when the image is enlarged, it is better to have a low maximum magnification. Conventionally, even when the drive range of the focus lens is limited, the magnification was determined in the same way as in full mode, resulting in an unnecessarily large magnification and a deterioration in image quality. In addition, unnecessary cropping occurs on the image to which the reference magnification is applied.

For example, even when normal mode is set, the magnification based on full mode is conventionally applied, so an image captured at a focal distance of 0.5 m, the closest end of the focus lens's driving range, is enlarged by 130% (1.3x). Furthermore, to match the angle of view of an image captured at a focal distance of infinity with the angle of view when the focal distance is 0.5 m, an additional 115% (1.15x) is enlarged. In other words, an overall magnification of 1.5x (1.3x x 1.15x) is applied to an image captured at a focal distance of infinity.

In contrast, in this embodiment, the correction magnification for an image captured at a focal distance of 0.5 m, which is the closest end of the focus lens's driving range, is used as the reference magnification. Therefore, no enlargement processing is applied to images captured at a focal distance of 0.5 m (although 100% (1x) enlargement processing may be applied). Therefore, the total magnification of the enlargement processing applied to correct angle-of-view fluctuations for images captured at infinity is 1.15x, which makes it possible to suppress degradation of image quality due to enlargement.

This embodiment assumes a lens device in which breathing causes the angle of view to widen as the focal length increases. Therefore, the correction magnification at the focus lens position corresponding to the shortest focal length at which the angle of view is smallest within the driving range of the focus lens is set to the reference magnification. This embodiment can suppress changes in the angle of view due to breathing, while also suppressing degradation of image quality due to correction of the angle of view more than conventional methods.

Using Figure 4, we will explain how to obtain the correction magnification from the reference magnification and maximum magnification according to the focus limit switch. The focus limit switch can be set to either full mode, macro mode, or normal mode, as in Figure 3.

FIG. 4 corresponds to an example in which the reference magnification and maximum magnification are set as follows according to the state of the focus limit switch.
Full mode: Standard magnification is 100% correction magnification, maximum magnification is 150%
Macro mode: Standard magnification is 100% and maximum magnification is 130%
Normal mode: Standard magnification is 130% of the correction magnification, and the maximum magnification is 150%

4 shows an example of the relationship between the position of the focus lens and the correction magnification. The relationship shown by the solid line 201 is the same as that shown in FIG.
Reference numeral 402 denotes the correction magnification at the focus lens position corresponding to a focal distance of 0.3 m.
Reference numeral 403 denotes the correction magnification at the focus lens position corresponding to a focal distance of 0.5 m.
Reference numeral 404 denotes the correction magnification at the focus lens position corresponding to the infinity focal distance.

If the driving range of the focus lens is not taken into consideration, as shown in FIG. 2, 402 is 100%, 403 is 130%, and 404 is 150%.
In full mode, the focus lens is driven over the entire movable range. Therefore, as in Figure 2, a correction magnification of 100% at a focus lens position corresponding to a focal distance of 0.3 m is used as the reference magnification, and magnification processing is applied up to a maximum magnification of 150% at a focus lens position corresponding to a focal distance of infinity. This corresponds to the image magnification correction processing shown in Figure 3(a).

In macro mode, the focus lens is driven within a range corresponding to a focus range of 0.3 to 0.5 m. In this case, a correction magnification of 100% at the focus lens position corresponding to a focus distance of 0.3 m is used as the reference magnification, and enlargement processing is applied up to a maximum magnification of 130% at the focus lens position corresponding to a focus distance of 0.5 m. This corresponds to the same image magnification correction processing as shown in Figure 3 (b).

In normal mode, the focus lens is driven within a range corresponding to a focus range of 0.5 to infinity. In this case, a correction magnification of 130% at a focus lens position corresponding to a focus distance of 0.5 m is used as the reference magnification, and correction is performed up to a maximum magnification of 150% at a focus lens position corresponding to a focus distance of infinity. By changing the correction magnification of 130% in full mode to the reference magnification of 100%, the maximum magnification becomes 115%. This corresponds to the image magnification correction process shown in Figure 3(c).

The system control unit 130 can determine the angle of view correction value to be set in the image magnification correction unit 142 as follows.
The correction magnification corresponding to the focus lens position (focusing distance) is set as the angle of view correction value, and the angle of view correction value is determined as follows.
When the angle of view correction value is less than the reference magnification: the angle of view correction value is set to 100%.
When the angle of view correction value is equal to or greater than the reference magnification and less than the maximum magnification: angle of view correction value=angle of view correction value/reference magnification×100%.
When the angle of view correction value is greater than the maximum magnification: angle of view correction value=maximum magnification/reference magnification×100%.

For example, the angle of view correction value for normal mode is 130% for the reference magnification (focusing distance 0.5 m) and 150% for the maximum magnification (focusing distance infinity). The correction magnification (130%) of 403, which corresponds to a focusing distance of 0.5 m, is greater than or equal to the reference magnification (130%), so the angle of view correction value is 130/130 x 100%, or 130%. Furthermore, the correction magnification (150%) of 404, which corresponds to a focusing distance infinity, is equal to the maximum magnification, so the image correction value is 150/130 x 100%, or 115%. This is equivalent to the relationship shown in Figure 3(c).

In addition, even when the focus range is set to full mode, macro mode, or normal mode by the settings of the imaging device 100, the system control unit 130 can determine the angle of view correction value in the same way as when the mode is set by the focus limit switch.

If the lens device 150 does not have a focus limit switch, the system control unit 130 acquires information related to image magnification correction stored in the lens device 150. The system control unit 130 also acquires the focus lens position (focus distance), reference magnification, and maximum magnification. The system control unit 130 can then determine the angle of view correction value as follows:

When the angle of view correction value is smaller than the reference magnification: the angle of view correction value is set to 100%.
When the angle of view correction value is equal to or greater than the reference magnification and less than the maximum magnification: angle of view correction value=angle of view correction value/reference magnification×100%.
When the angle of view correction value is equal to or greater than the maximum magnification: angle of view correction value=maximum magnification/reference magnification×100%.
In this way, the system control unit 130 can determine the angle of view correction value by the same process whether the lens device 150 has a focus limit switch or not.

The reference magnification and maximum magnification may be changed in response to user instructions from the camera operation unit 117 using information stored in the lens device 150.

In the example shown above, we described a configuration in which, when the drive range of the focus lens is limited on the close-up side, the reference magnification is changed so that the correction magnification corresponding to the close-up end of the limited drive range is 100% (1x). However, when the drive range of the focus lens is limited on the close-up side, the effect of the present invention of suppressing the maximum magnification can be achieved by making the correction magnification corresponding to the close-up end of the drive range smaller than in full mode (but still 100% or greater).

Next, the procedure of the image magnification correction process described above will be described with reference to the flowchart shown in FIG.
In S501, the system control unit 130 acquires information related to image magnification correction for the attached lens device 150, and stores the information in a non-volatile area of the memory 107 in association with, for example, the model name of the lens device 150. S501 can be executed when the imaging device 100 is started up, when attachment of the lens device 150 is detected, when a predetermined user instruction is detected, etc. Note that if information related to image magnification correction for the lens device 150 is already stored in the memory 107, S501 does not need to be executed.

Figure 7 is a diagram showing an example of the data structure of data (image magnification correction data) related to image magnification correction stored in memory 107. The image magnification correction data has a basic information area 701, a header area 702, and a correction value storage area 703. In this embodiment, when the lens device 150 is a zoom lens with a variable focal length, the focal length range between the wide end and the telephoto end is divided into multiple parts as information related to image magnification correction. Then, the reference magnification and maximum magnification are stored for combinations of focal lengths at the division points and the state of the focus limit switch. In addition, the focusable distance range is divided into multiple parts, and a correction magnification is stored for each combination of the focal length and the focus distance corresponding to the division points.

The basic information area 701 stores information indicating whether the lens device supports image magnification correction and information indicating the number of correction information items. By referencing the basic information area 701, the system control unit 130 can determine whether the attached lens device 150 supports image magnification correction. If the system control unit 130 determines that the attached lens device 150 does not support image magnification correction, it may perform shooting without performing image magnification correction.

The header area 702 stores information such as the number of divisions (nNum) of the focal length range of the lens device 150 (multiple in the case of a zoom lens), the number of divisions of the focusable distance range (fNum), the number of states of the focus limit switch (swNum), and information on the division points of each piece of information. swNum is a natural number greater than or equal to 1, and an swNum of 1 indicates that the lens device does not have a focus limit switch.

Discrete values of the focal length are stored in the focal length n[0] to n[nNum-1].
The focal lengths f[0][0] to f[nNum-1][fNum-1] store discrete focal length values for each discrete focal length.
The reference magnifications b[0][0] to b[nNum-1][swNum-1] store the correction magnifications that become the reference magnification for each combination of the focus limit switch state and a discrete focal length.
The maximum magnification m[0][0] to m[nNum-1][swNum-1] stores the correction magnification that becomes the maximum magnification for each combination of the focus limit switch state and the discrete focal length.

The correction value storage area 703 stores a correction magnification for each combination of a discrete focal length and a discrete in-focus distance. This correction magnification corresponds to the correction magnification in full mode.

The system control unit 130 can obtain the reference magnification and maximum magnification from the header area 702 based on the state of the focus limit switch at the time of shooting and the focal length of the lens device 150.

In addition, the system control unit 130 can obtain the corresponding correction magnification from the correction value storage area 703 based on the focal length and in-focus distance (focus lens position) of the lens device 150 at the time of shooting.

For information corresponding to value combinations that are not stored, the system control unit 130 can obtain it by linearly interpolating stored value combinations that are close to each other.

The total amount of information stored can be determined from the number of divisions (number of pieces of information) of the focal length range, the number of divisions of the focusable distance range, and the number of states of the focus limit switch stored in the header area 702. Furthermore, if the data stored in the header area 702 and the correction value storage area 703 is of a fixed length, it is easy to determine the address where the required information is stored.

Instead of the focal length n, the number of variable aberration information divisions sNum, the variable aberration information s, and the corresponding image magnification may be used. When a lens device 150 that stores variable aberration information is attached, the imaging device 100 performs image magnification correction processing according to the variable aberration amount.

Once the information necessary for image magnification correction is prepared in memory 107, the system control unit 130 waits for detection of turning on SW2 or operation of the video recording start button while executing processing in the shooting standby state. Processing in the shooting standby state is, for example, live view display processing or processing in response to user operation other than on the shutter buttons 115 and 116.

In S502, the system control unit 130 starts the shooting operation when it detects that SW2 has been turned on or the video shooting button has been operated.

In S503, the system control unit 130 performs processes such as automatic exposure control (AE) and autofocus detection (AF). The details of the processes may differ between still image capture and video capture.

In S504, the system control unit 130 controls the operation of the image sensor 102 and other components to capture a still image or one frame of video. The image generation unit 103 also generates a digital image signal from the analog image signal read from the image sensor 102.

In S505, the system control unit 130 acquires information (shooting information) relating to the state of the lens device 150 at the time of shooting in S504 from the lens control unit 160. The shooting information includes focal length and in-focus distance (focus lens position), and may also include variable aberration amounts depending on the model of the lens device 150. The system control unit 130 stores the acquired shooting information in memory 107.

In S506, the system control unit 130 acquires the state of the focus limit switch of the lens device 150 from the lens control unit 160 and stores it in the memory 107. Note that the state of the focus limit switch may also be acquired as shooting information in S505.

In S507, the system control unit 130 determines a field of view correction value from the information related to image magnification correction acquired in S501, the shooting information acquired in S505, and the state of the focus limit switch acquired in S506, and sets the value in the image magnification correction unit 142.

In S508, the system control unit 130 controls the image processing unit 140 to apply image processing, including image magnification correction processing, to the one frame of image acquired in S504. The image processing unit 140 generates a still image data file or video data file for recording, and image data for display. The image data for display is stored in memory 107, and the image data file for recording is recorded in the recording unit 108.

In S509, the system control unit 130 determines whether the conditions for ending shooting are met. For example, if SW2 remains on during still image shooting, or if operation of the video shooting button is not detected during video shooting, the system control unit 130 determines that the conditions for ending shooting are not met and executes S503 to shoot the next frame. On the other hand, for example, if SW1 and SW2 are off during still image shooting, or if operation of the video shooting button is detected during video shooting, the system control unit 130 determines that the conditions for ending shooting are met, and ends the processing shown in FIG. 5.

Next, the communication operation performed between the lens device 150 and the imaging device 100 will be described using the flowchart shown in Figure 6. Communication between the lens device 150 and the imaging device 100 is initiated by either the lens device 150 or the imaging device 100.

In S601, the lens control unit 160 sends basic information to the system control unit 130. The basic information may be static information about the lens device 150 stored in the lens device 150, such as model information, whether or not image magnification correction information is present, whether or not a focus limit switch is present and the number of states it is in. Note that the system control unit 130 may also send model information about the imaging device 100 to the lens control unit 160. S601 may be executed, for example, when the imaging device 100 is started up or when the lens device 150 is attached to the imaging device 100.

In S602, the lens control unit 160 determines whether a request for information regarding image magnification correction has been received from the system control unit 130, and if it is determined that a request has been received, it executes S603; if not, it executes S604.

In S603, the lens control unit 160 transmits information related to image magnification correction stored in the lens information storage unit 165, i.e., the image magnification correction data shown in FIG. 7, to the system control unit 130. Then, the lens control unit 160 executes S604.

In S604, the lens control unit 160 determines whether a request for the state of the focus limit switch has been received from the system control unit 130, and if it is determined that a request has been received, it executes S605; if not, it executes S606.

In S605, the lens control unit 160 sends information indicating the state of the focus limit switch to the system control unit 130. The lens control unit 160 then executes S606.

In S606, the lens control unit 160 determines whether a request for shooting information has been received from the system control unit 130, and if it is determined that a request has been received, it executes S607; if not, it executes S608.

In S607, the lens control unit 160 transmits information regarding the current focal length and in-focus distance to the system control unit 130 as shooting information. The lens control unit 160 then executes S608.

In S608, the lens control unit 160 determines whether a communication stop instruction has been received from the system control unit 130, and if it is determined that a communication stop instruction has been received, it ends communication with the system control unit 130; if not, it executes S602.

Here, we have described a case where angle-of-view variation correction is performed before recording. However, angle-of-view variation correction can also be performed after recording. For example, the image data for recording can be recorded including information related to image magnification correction shown in Figure 7 or information that can identify information related to image magnification correction (such as the model name of the lens device), as well as shooting information for each frame (including the state of the focus limit switch). A device that processes recorded image data can obtain information related to image magnification correction from an image data file or an external device, and use this information together with the shooting information to correct angle-of-view variation in the recorded image data.

(Variation 1)
When the focus range is changed during continuous shooting of still images or video shooting, it may be determined whether to change the reference magnification depending on the focus range before and after the change. For example, when switching from full mode to normal mode, the reference magnification may not be changed when switching to normal mode, and correction may be continued in full mode so that the angle of view does not change midway.

(Variation 2)
In this embodiment, the correction of angle of view variation for a lens apparatus in which the angle of view widens at infinity has been described. However, similar correction is possible for a lens apparatus in which the angle of view widens at the closest end. In this case, in the relationships shown in FIGS. 2 to 4, the closest end and infinity positions of the focus lens position are reversed, and the magnification becomes the trimming ratio. If the maximum distance of the focus range is limited to a distance closer than infinity, the trimming ratio of the focus lens position at the infinity end of the limited focus range can be changed to 100% (no trimming). This prevents unnecessary trimming.

(Variation 3)
In this embodiment, an example has been described in which the lens device 150 transmits the image magnification correction data to the image capturing device 100, but the image capturing device 100 may be configured to acquire the image magnification correction data by another method.

For example, when an adapter device is connected between the lens device 150 and the imaging device 100, the imaging device 100 may acquire image magnification correction data for the lens device 150 from the adapter device. For example, if the adapter device has image magnification correction data for a lens device of a different type than the lens device 150, it may receive information indicating the type of lens device attached from the lens device 150 or the imaging device 100. The adapter device may then be configured to transmit image magnification correction data for the lens device type corresponding to the received information to the imaging device 100. Examples of adapter devices include mount converters that convert mount types, extenders with optical systems, and speed boosters. Devices that can be directly attached to the imaging device 100, such as adapter devices and lens devices 150, are collectively referred to as accessory devices.

(Variation 4)
In the present embodiment, an example has been described in which multiple corrections for each condition (e.g., the shooting state of the lens device 150) are stored in the image magnification correction data, and the imaging device 100 identifies a correction value by comparing the image magnification correction data with the shooting conditions. Here, a configuration may be adopted in which the lens device 150 performs a process of comparing the image magnification correction data with the shooting conditions and transmits the identified correction value to the imaging device 100.

(Variation 5)
Furthermore, some or all of the configuration of this embodiment may be performed on a server. For example, the imaging device 100 may download image magnification correction data for the lens device 150 stored in storage on the server, rather than in the lens device 150, from the server. Alternatively, the imaging device 100 may transmit the type of lens 150 and the conditions at the time of shooting to the server, and download correction values corresponding to the conditions at the time of shooting from the server. Alternatively, the captured still image data or video data and the corresponding conditions at the time of shooting may be uploaded to the server, image processing may be performed by a processing unit on the server, and the corrected still image or video may be obtained from the server.

(Variation 6)
The image magnification correction of this embodiment may also be performed by an image processing device such as a PC. In this case, the image processing device acquires image magnification correction data from a server or the like, and acquires still image data or video data directly or indirectly from the imaging device 100, thereby performing image magnification correction.

In this embodiment, information regarding image magnification correction according to the state of the components that set the focus range of the lens device is obtained, and the angle of view of an image captured using the lens device is corrected based on the information regarding image magnification correction and information regarding the state of the lens device at the time the image was captured. For example, if the drive range of the focus lens of the lens device is limited, the reference magnification used in image enlargement processing to correct changes in angle of view that occur as the focus lens moves can be changed according to the drive range. In this case, changes in angle of view can be corrected without unnecessarily enlarging the image, and degradation of image quality due to correction can be suppressed.

● (Second embodiment)
Next, a second embodiment of the present invention will be described. This embodiment relates to white balance adjustment processing in the image processing unit 140. When trimming is performed to correct breathing, if the area removed by trimming is reflected in the white balance adjustment processing, the color tone of the image after trimming may change unnaturally.

This embodiment addresses this issue by achieving appropriate white balance adjustment processing even when performing image magnification correction processing.

FIG. 8 is a flowchart relating to the white balance control process when the image magnification correction process is performed.
In S901, the image processing unit 140 acquires shooting information for a frame to be processed. Note that the shooting information may be supplied to the image processing unit 140 when it is acquired by the system control unit 130.
In S902, the image processing unit 140 acquires the maximum magnification for image magnification correction from the memory 107 based on the state of the focus limit switch included in the shooting information or acquired separately. For example, if the state of the focus limit switch in the lens apparatus described in the first embodiment is the normal mode, 115% is acquired as the maximum magnification.

In S903, the image processing unit 140 uses the maximum magnification to determine the reduction ratio R of the detection area to be used in the WB adjustment process. The detection area may be a detection area for calculating a WB correction value for pixels estimated to be white, a detection area for calculating a WB correction value for a specific subject (e.g., natural green), or a characteristic area such as a face related to WB control.

The image processing unit 140 can determine the reduction ratio R of the detection area using the maximum magnification Emax acquired in S902, for example, according to the following equation (1).
R=1/Emax (1)
For example, when the focus limit switch is in the normal mode, the maximum magnification is 115%, so the reduction ratio of the detection area is determined to be 87%.

In S904, the image processing unit 140 sets the detection area to be used for WB adjustment processing using the reduction ratio determined in S903. Specifically, the image processing unit 140 reduces the detection area to be used for WB control when image magnification correction processing is not performed so that the reduction ratio becomes the one calculated in S903.

In S905, the image processing unit 140 uses the detection area set in S904 to calculate the WB correction value in the same way as when image magnification correction processing is not performed. Note that the process of calculating the WB correction value using an area of white pixels or pixels of a specific color from an image as the detection area is well known, so a detailed explanation will be omitted.

As such, according to this embodiment, when cropping is performed to correct changes in the angle of view due to lens breathing, the area removed by the cropping process is not reflected in the white balance adjustment process. This allows the white balance adjustment process to be performed based on the corrected image area visually recognized by the user, and reduces changes in color tone due to the influence of the area removed by the cropping process.

Note that while the white balance adjustment process has been described here, the white balance correction value is one example of an image-based evaluation value. Similar effects can be achieved for other image-based evaluation values by avoiding the use of areas removed by the trimming process.

(Other embodiments)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The present invention is not limited to the contents of the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

100...imaging device, 102...imaging element, 130...system control unit, 140...image processing unit, 142...image magnification correction unit, 150...lens device, 151...focus lens, 160...lens control unit, 161...lens operation unit, 165...lens information storage unit

Claims (12)

an acquisition unit that acquires, from a lens device, information regarding image magnification correction according to a state of a member that sets a focus range of the lens device;
a correction unit that corrects the angle of view of an image captured using the lens device based on information related to the image magnification correction and information related to the state of the lens device at the time the image was captured;
and
The imaging device according to claim 1, wherein the information about the state of the lens device at the time of shooting includes the state of the member. the information regarding the image magnification correction includes a reference magnification and a maximum magnification for each state of the member;
2. The imaging device according to claim 1, wherein the correction means corrects the angle of view of the image based on the reference magnification and the maximum magnification according to the state of the member at the time of photographing. the information about the state of the lens device at the time of photographing includes a focal length;
3. The imaging device according to claim 2, wherein the correction means corrects the angle of view of the image using a magnification ranging from the reference magnification to the maximum magnification according to the state of the member and the focal distance. the focal length of the lens device is variable, and the information regarding the image magnification correction is information corresponding to a combination of the state of the member and the focal length;
the information about the state of the lens device at the time of photographing includes a focal length;
3. The imaging device according to claim 2, wherein the correction means corrects the angle of view of the image based on the reference magnification and the maximum magnification according to a combination of the state of the component at the time of shooting and the focal length. The imaging device described in claim 4, characterized in that the correction means corrects the angle of view by enlarging the image by a magnification of 1, which is a correction magnification corresponding to the reference magnification corresponding to the combination of the focal length and the state of the component at the time of shooting, among the correction magnifications corresponding to the combination of the focal length and the focal distance of the lens device. The imaging device described in claim 5, characterized in that the acquisition means further acquires the correction magnification from the lens device. an acquisition step in which an acquisition unit acquires, from a lens device, information regarding image magnification correction according to a state of a member that sets a focus range of the lens device;
a correction step in which a correction unit corrects the angle of view of an image captured using the lens device based on information related to the image magnification correction and information related to the state of the lens device at the time of capturing the image;
and
The control method for an imaging device, wherein the information about the state of the lens device during imaging includes the state of the components. A lens device that is detachable from an imaging device,
a storage means for storing information regarding image magnification correction according to the state of a member for setting the focus range of the lens device;
a transmitting unit for transmitting information relating to the image magnification correction to an imaging device to which the lens device is attached. A lens device according to claim 8, characterized in that the information regarding the image magnification correction includes a reference magnification and a maximum magnification for each state of the member. the transmitting means further transmits information about the state of the lens device to the imaging device in response to a request from the imaging device;
10. The lens device according to claim 8, wherein the information relating to the state of the lens device includes a focal length and the state of the member. A lens device according to claim 8 or 9, characterized in that the focal length of the lens device is variable, and the information regarding the image magnification correction is information corresponding to a combination of the state of the component and the focal length. The lens device of claim 10, wherein the focal length of the lens device is variable, and the information about the state of the lens device further includes the focal length.