JP7094767B2 - Imaging device and its control method and program - Google Patents

Description

The present invention relates to an image pickup apparatus, a control method thereof, and a program thereof, and more particularly to an image pickup apparatus that performs focus detection by an image pickup surface phase difference detection method.

An image pickup device including an image pickup device that acquires an image for recording or display and performs focus detection by an image pickup surface phase difference detection method is known. For example, the image pickup device described in Patent Document 1 has two image pickup elements that view the same exit pupil, and these two image pickup elements include a plurality of pixel portions for focal detection having different baseline lengths from each other. Further, in Patent Document 1, when the optical diaphragm (F value) becomes a predetermined value or more while the focus detection process is performed by one image sensor, the focus is detected by the other image sensor. It is described that the reduction of the focus detection ability due to the vignetting of the luminous flux is suppressed by switching.

Japanese Unexamined Patent Publication No. 2016-224208

However, in the technique described in Patent Document 1, since the optical exit pupils used for photography and focus detection are the same, if the F value setting suitable for each of photography and focus detection is different, the distance measurement accuracy is lowered. It may end up.

An object of the present invention is to provide an imaging device capable of performing imaging while maintaining distance measurement accuracy regardless of imaging conditions.

The image pickup apparatus according to the present invention passes through different pupil regions of the first optical system, the first image pickup pixel in which the light beam passing through the emission pupil of the first optical system is formed, and the first optical system. It has a main optical unit having a main image pickup element having a second image pickup pixel on which the light beam is formed, a second optical system, a third optical system, and a sub image pickup element. The sub-optical unit in which the light beam passing through the optical system and the light beam passing through the third optical system form an image in different regions of the sub-imaging element, and the output signal from the main image pickup element or the sub-imaging element. The generation means for generating a phase difference signal by using the main optical unit and the sub-optical unit are controlled, and the output signal from the main image pickup element and the sub-image pickup element are controlled according to the control state of the main optical unit. A control means for selecting one of the output signals to measure the distance to the subject is provided , and the first optical system, the second optical system, and the third optical system are independent optical systems. It is characterized by being.

According to the present invention, it is possible to perform imaging while maintaining the distance measurement accuracy regardless of the imaging conditions.

It is external perspective view of the image pickup apparatus which concerns on embodiment of this invention. It is a block diagram of an image pickup apparatus. It is a figure explaining the image pickup region of the sub-imaging optical system of an image pickup apparatus. It is a figure explaining the structure of the image pickup pixel in the main image pickup element of an image pickup apparatus. It is a figure explaining the structure of the focus detection pixel in the main image pickup element of an image pickup apparatus. It is a figure explaining the pixel arrangement of the main image pickup element of an image pickup apparatus. It is a schematic diagram explaining the incident light flux to the main image pickup element of an image pickup apparatus. It is a figure explaining the distance measuring principle by the sub-imaging optical system of an image pickup apparatus. It is a flowchart explaining the image pickup operation of the image pickup apparatus. It is a flowchart explaining the process of S504 in 1st Embodiment. It is a flowchart explaining the process of S504 in 2nd Embodiment. It is a figure explaining the pixel composition of the main image pickup element in 3rd Embodiment. It is a figure explaining the light-receiving state in the main image sensor in 3rd Embodiment. It is a figure explaining the pixel circuit composition of the main image pickup element in 3rd Embodiment. It is a figure explaining the driving method of the main image pickup element in 3rd Embodiment. It is a flowchart of the focus detection operation in the image pickup apparatus which concerns on 3rd Embodiment. It is a figure explaining the reason which sets the threshold value in the output of the main image pickup element in 3rd Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
In the first embodiment, when vignetting of the luminous flux due to the optical diaphragm occurs during distance measurement by the image pickup surface phase difference method using the main image sensor, distance measurement is performed using the sub-image sensor having a plurality of image regions. The image pickup device for performing the above will be described.

FIG. 1A is an external perspective view of the image pickup apparatus 1 according to the first embodiment. Specifically, the image pickup device 1 is an electronic still camera having a moving image shooting function or a video camera having a still image shooting function. The image pickup apparatus 1 has a main image pickup optical system 10 (main optical unit) including a photographing lens and a sub image pickup optical system 20 (secondary optical unit) including two photographing lenses on the front surface of the housing. The main imaging optical system 10 is mainly used for capturing still images and moving images for recording. The sub-imaging optical system 20 is used for external measurement AF (Auto Focus) and external measurement AE (Auto Exposure) for assisting photography by the main imaging optical system 10.

FIG. 1B is an external perspective view of an automobile equipped with the image pickup apparatus 2 according to the first embodiment. The image pickup device 2 is a so-called in-vehicle camera, and has a main image pickup optical system 10A and a sub image pickup optical system 20A on the front surface of the vehicle body. The main imaging optical system 10A is used for distance measurement and acquisition of an object detection image. The sub-imaging optical system 20A is used for external measurement AF and external measurement AE.

The main image pickup optical system 10 of the image pickup device 1 and the main image pickup optical system 10A of the image pickup device 2 have the same applications and functions, and the sub image pickup optical system 20 of the image pickup device 1 and the sub image pickup optical system 20A of the image pickup device 2 Since the uses and functions are the same, the description will be continued below by taking the image pickup apparatus 1 as an example.

FIG. 2 is a block diagram of the image pickup apparatus 1. The image pickup apparatus 1 includes a main image pickup optical system 10, a main image pickup signal processing unit 11, a main phase difference processing unit 12, a compression / expansion unit 13, a control unit 14, a light emitting unit 15, an operation unit 16, an image display unit 17, and an image storage unit. 18 is provided. Further, the image pickup apparatus 1 includes a sub-imaging optical system 20, a sub-imaging signal processing unit 204, and a sub-phase difference processing unit 205. In FIG. 2, the broken line arrow represents the reflected light (incident light) from the subject, the alternate long and short dash line arrow represents the control signal and the calculated value after the calculation process, and the solid line arrow represents the flow of the image signal.

The main image pickup optical system 10 has a main optical lens barrel 101 as a first optical system and a main image pickup element 102. The primary optical lens barrel 101 includes a lens for condensing light from a subject on the main image pickup element 102, a focus mechanism unit 1011 for adjusting the focus, and a zoom mechanism unit 1012 for varying the magnification of optical imaging. .. Further, the main optical lens barrel 101 has a diaphragm mechanism unit 1013 for adjusting the amount of light incident on the main image sensor 102, and a shutter mechanism unit 1014. The focus mechanism unit 1011, the zoom mechanism unit 1012, the aperture mechanism unit 1013, and the shutter mechanism unit 1014 are driven according to the control signal from the control unit 14.

The main image sensor 102 is composed of an image pickup pixel (first image pickup pixel) described later, a pixel portion composed of a focus detection pixel (second image pickup pixel (pixel for phase difference detection)), and an AD converter (not shown). Includes XY readout type CMOS image sensors and the like. The main image sensor 102 outputs an image signal by executing an image pickup operation such as exposure, signal reading, and reset according to a control signal from the control unit 14.

The sub-imaging optical system 20 includes a sub-optical lens barrel 201 as a second optical system and a sub-optical lens barrel 202 as a third optical system arranged at predetermined intervals so that the optical axes are substantially parallel to each other. It has an auxiliary image pickup element 203. Similar to the primary optical lens barrel 101, the sub-optical lens barrels 201 and 202 each have a built-in focus mechanism unit, zoom mechanism unit, and aperture mechanism unit (not shown). Since the sub-imaging optical system 20 is mainly used for external measurement AF and external measurement AE processing, it is sufficient that the imaging state of the main imaging optical system 10 can be grasped from the optical image obtained by the sub-imaging optical system 20. Therefore, the focus distance of the secondary optical lens barrels 201 and 202, the angle of view by the zoom setting, the aperture setting, etc. may be set to conditions suitable for AF and AE processing, and the shooting conditions set in the primary optical lens barrel 101 are not necessarily the same. Does not have to be the same as.

The light collected by the sub-optical lens barrels 201 and 202 from the subject is irradiated in the plane of the sub-image sensor 203 at intervals. Similar to the main image sensor 102, the sub image sensor 203 uses an XY readout type CMOS image sensor or the like. The sub-image sensor 203 outputs an image signal by executing an image pickup operation such as exposure, signal reading, and reset according to a control signal from the control unit 14. Unlike the main image sensor 102, the sub image sensor 203 does not have a focus detection pixel. The sub-image sensor 203 images each subject image by the sub-optical lens barrels 201 and 202 with ordinary image pickup pixels, and parallax detection is performed using the obtained image pickup signal as described later.

Here, with reference to FIG. 3, the imaging region of the light emitted to the sub-image sensor 203 will be described. FIG. 3A is a schematic view showing a cross-sectional view of the sub-imaging optical system 20. As shown in FIG. 3A, the light from the subject is collected by the sub-optical lens barrels 201 and 202, respectively, and is irradiated on the imaging surface of the sub-imaging device 203 at a distance from each other. FIG. 3B is a plan view illustrating an imaging region of light (infinity subject image) irradiated on the imaging surface of the sub-imaging element 203. The subject images emitted from the sub-optical lens barrels 201 and 202 correspond to the image circle 301 corresponding to the image of the sub-optical lens barrel 201 and the image corresponding to the image of the sub-optical lens barrel 202 with respect to the image plane of the sub-image sensor 203. The subject image is projected on the area of the circle 302. The light collected by the sub-optical lens barrel 201 forms an image on the image circle 301 as the first imaging region, so that the subject image is projected in the image circle 301. Similarly, the light collected by the sub-optical lens barrel 202 forms an image on the image circle 302 as the second imaging region, so that the subject image is projected in the image circle 302. In this way, by separately projecting the image circles of the two optical systems on one image sensor, the parallax signal of the subject can be obtained from the output signal of one image sensor.

Although the details will be described later, the sub-phase difference processing unit 205 detects the parallax signal using the image data of the image regions 303 and 304 in the image circles 301 and 302. The details of the parallax detection method using the image data obtained from the sub-image sensor 203 will be described later. Further, in the first embodiment, one sub-image sensor is provided for each of the two optical systems, but an image sensor is arranged in each of the two optical systems, and the output from each image sensor (image sensor). May be used to obtain a parallax signal. Further, although the image pickup apparatus 1 is individually provided with the main phase difference processing unit 12 and the sub-phase difference processing unit 205 (FIG. 2), it may be provided as a common phase difference processing unit.

Returning to the description of the block configuration of the image pickup apparatus 1 with reference to FIG. Under the control of the control unit 14, the main image pickup signal processing unit 11 performs color correction processing, AE processing, white balance processing, optical shading processing, etc. of the image pickup signal acquired from the main image pickup element 102, and the image signal is generated. The control signal is output to the control unit 14. The control unit 14 temporarily stores the image signal and the control signal acquired from the main image pickup signal processing unit 11 in the RAM in the control unit 14.

The main phase difference processing unit 12 acquires an image pickup surface phase difference image of a different pupil surface from the main image pickup element 102, and performs a known phase difference detection process from the image signal of the acquired image pickup surface phase difference image to perform a phase difference. Generate a signal. Under the control of the control unit 14, the compression / decompression unit 13 takes out an image signal stored in the RAM in the control unit 14 from the main image pickup signal processing unit 11 and compresses and encodes it in a predetermined still image data format such as a JPEG method. Perform processing. Further, the compression / decompression unit 13 performs decompression / decoding processing of the coded data of the still image supplied from the control unit 14. The compression / decompression unit 13 may be capable of performing compression coding / decompression / decoding processing of a moving image by an MPEG method or the like.

The control unit 14 is, for example, a microcontroller composed of a CPU, ROM, RAM, etc., and controls each unit of the image pickup apparatus 1 in an integrated manner by expanding (executing) a program stored in the ROM or the like into the RAM. do. The light emitting unit 15 is a strobe device or an LED light emitting device using a xenon tube, and irradiates the subject with light when it is determined by the AE processing in the main imaging signal processing unit 11 that the exposure value of the subject is low. do. The operation unit 16 is composed of, for example, various operation keys such as a shutter release button, a lever, a dial, and the like, and outputs a control signal corresponding to an input operation by the user to the control unit 14.

The image display unit 17 is composed of a display device such as an LCD, an interface circuit for the display device, and the like, and generates an image signal to be displayed on the display device based on the image signal supplied from the control unit 14, and the generated image. Supply the signal to the display device. As a result, the image is displayed on the display device. The image storage unit 18 is composed of, for example, a portable semiconductor memory, an optical disk, an HDD, a magnetic tape, or the like, and receives and stores image data encoded by the compression / decompression unit 13 from the control unit 14. Further, the image storage unit 18 reads out the designated data according to the control signal from the control unit 14 and outputs the data to the control unit 14.

The sub-image sensor processing unit 204 acquires an image signal from the sub-image sensor 203 and performs the same processing as the main image sensor 11 to generate AE information and the like necessary for image pickup control of the sub-image sensor 203. .. The sub-parallax processing unit 205 performs known phase difference processing using the image signals of the image regions 303 and 304 imaged by the sub-image sensor 203, so that the captured image obtained through the sub-optical lens barrels 201 and 202 can be obtained. Find the parallax. The details of the parallax detection method will be described later.

FIG. 4A is a plan view illustrating an image pickup pixel (a pixel for obtaining a captured image (photograph)) of the main image pickup element 102. 4 (b) is a cross-sectional view taken along the line AA shown in FIG. 4 (a), and shows a configuration in the vicinity of the image pickup pixel. In the main image sensor 102, of the 4 pixels of 2 × 2, pixels having a spectral sensitivity of green (G) are arranged in two pixels located diagonally, and red (R) and blue (R) and blue (in the other two pixels) are arranged in the other two pixels. It has a Bayer array in which pixels having the spectral sensitivity of B) are arranged, and a structure of 2 rows × 2 columns is repeatedly arranged. In the main image sensor 102, focus detection pixels, which will be described later, are distributed and arranged between Bayer arrays according to a predetermined rule. The pixel arrangement in the sub-image sensor 203 is the same as the pixel arrangement in the main image sensor 102. However, the sub-image sensor 203 does not have a focus detection pixel.

Each image pickup pixel has an on-chip microlens ML, a color filter CF, a wiring layer CL for forming a signal line for transmitting a signal, and a photodiode PD which is a photoelectric conversion unit, from the front side to the back side. It has an arranged structure. Here, as the color filter CF, a red (R) color filter CF _R and a green (G) color filter CF _G are shown. In FIG. 4B, the image pickup optical system TL corresponding to the main image pickup optical system 10 is schematically shown on the front surface of the image pickup pixel.

The on-chip microlens ML and the photodiode PD of the image pickup pixel are configured to capture the luminous flux passing through the image pickup optical system TL as effectively as possible. That is, the exit pupil EP of the imaging optical system TL and the photodiode PD are coupled by the on-chip microlens ML, and the effective area of the photodiode PD is designed to be large. Although FIG. 4B shows the incident light flux of the pixel CFR, the pixels CFG and _CFB of the green ( _G ) and the blue ( _B ) also have the same structure. In this way, the diameter of the exit pupil EP corresponding to each of the RGB pixels _CFR , _CFG , and _CFB is increased so that the light flux from the subject can be efficiently captured, so that the S / of the image signal can be taken in. N is improved.

FIG. 5A is a plan view illustrating the arrangement of the focus detection pixels in the main image sensor 102. 5 (b) is a cross-sectional view taken along the line BB shown in FIG. 5 (a), and shows a configuration in the vicinity of the focus detection pixel. Note that FIG. 5 illustrates a portion where the focus detection pixels _SHA and _SHB are arranged at the positions where the red (R) and blue (B) image pickup pixels are arranged in the Bayer array. The configuration of the on-chip microlens ML and the color filter CF is the same as the configuration of the image pickup pixels shown in FIG. 4 (b). However, the focus detection pixels _SHA and _SHB each have a white (W) color filter CF _W.

The opening of the wiring layer CL is deviated in one direction with respect to the center line of the on-chip microlens ML in order to divide the pupil in the horizontal direction (horizontal direction) of the photographing lens by the image pickup element. Specifically, the opening _OPHA of the focus detection pixel SHA is biased to the right side, and receives the light flux that has passed through the exit pupil EPHA on the left side of the imaging optical system TL. Similarly, since the opening _OPHB of the focus detection pixel SHB is biased to the left side, the light flux passing through the exit pupil EPHB on the right side of the imaging optical system is received. Here, the subject image acquired by the pixel group in which the focus detection pixels _{SHA are regularly arranged in the horizontal direction is defined as the A image, and the subject image is acquired by the pixel group in which the focus detection pixels SHB are} regularly arranged in the horizontal direction. Let B be the subject image. Then, by detecting the relative positions of the A image and the B image, the defocus amount (defocus amount) of the subject image can be detected. When it is desired to detect the amount of focus shift in the vertical direction (vertical direction), the opening _OPHA of the focus detection pixel SHA is biased upward, and the opening _OPHB of the focus detection pixel SHB is biased downward, and the opening OPHA, The opening shape of the OPHB may be rotated by 90 degrees.

FIG. 6 is a plan view showing the pixel arrangement of the main image sensor 102, and covers a range of 8 rows (Y direction) and 8 columns horizontally (X direction) of the CMOS image sensor sensor viewed from the main image sensor 10 side. Shows. A Bayer arrangement is applied to the color filters, green (G) and blue (B) color filters are alternately arranged in order from the left for pixels in even rows, and red in order from left for pixels in odd even rows. The color filters (R) and green (G) are arranged alternately.

The focus detection pixels _SHA and _SHB are arranged so as to replace the image pickup pixels at the positions where the red (R) or blue (B) color filters are arranged, and are arranged at predetermined line spacings in the first embodiment. Has been done. Here, the photodiode PDs of the focus detection pixels SHA and _SHB are different in area and position in the X direction as compared with the photodiode _PDs of the image pickup pixels (see FIG. 5B). The arrangement density and line spacing of the focus detection pixels _SHA and _SHB are not limited to the pixel arrangement shown in FIG. From the row in which the focus detection pixels _SHA and _SHB are arranged, an image pickup signal can be obtained by performing interpolation correction on the focus detection pixels _SHA and _SHB using peripheral image pickup pixels.

Next, a method for obtaining the defocus conversion coefficient for calculating the defocus amount from the image shift amount and a change in the baseline length due to vignetting of the luminous flux will be described. FIG. 7A is a schematic view showing how the incident light flux to the pixel near the center of the main image sensor 102 is limited by the aperture of the imaging optical system at the position of the exit pupil surface 701. A luminous flux limited by some constituent members such as the lens holding frame 740 of the photographing lens in the main optical lens barrel 101 and the optical aperture 741 in the aperture mechanism unit 1013 is incident on the main image pickup element 102. In FIG. 7A, the positions 703 and 704 of the main image sensor 102 indicate the position of the main image sensor 102 for schematically showing the defocus amount 729, and the position 703 indicates the position of the planned image plane. ing. The broken line 705 indicates the optical axis, and the point 706 indicates the position of the optical axis on the main image sensor 102. The luminous fluxes 707 and 708 show an example of the luminous flux limited by the optical diaphragm 741, and the luminous flux 709 and 720 show the luminous flux not limited by the optical diaphragm 741. The luminous flux 723 and 724 indicate the phase detection luminous flux of the luminous flux 709 and 720, and the luminous flux 721 and 722 indicate the phase detection luminous flux of the luminous flux 707 and 708. Points 727 and 728 indicate the position of the center of gravity of the luminous flux 723 and 724, and points 725 and 726 indicate the position of the center of gravity of the luminous flux 721 and 722.

FIG. 7B is a diagram illustrating a change in the position of the center of gravity due to vignetting of the light flux on the exit pupil surface 701 of the focus detection pixels _{SHA and SHB in} the central portion of the main image sensor 102. The pupil region 733 shows the pupil region of the limited luminous flux 707,708 with respect to the central pixel of the main image sensor 102. Further, the pupil region 734 shows the pupil region of the luminous flux 709,720 which is not limited to the central pixel of the main image sensor 102. The image 735 shows the incident angle characteristic of the image of the focus detection pixel _{SHA (see FIG. 5), and the image 736 shows the incident angle characteristic of the image of the focus detection pixel SHB (} see FIG. 5).

Luminous flux transmitted through the inside of the shape shown in the pupil regions 733 and 734 is incident on the focus detection pixels SHA and _SHB with the sensitivity distribution shown in the images 735 and ₇₃₆ , respectively. The sensitivity distribution is shown by shading, and is strong (dark) on the center side and weak (light) on the outer peripheral side. By obtaining the distribution center of gravity of each phase detection light flux transmitted through the circle indicating the pupil region 733, 734, the center-of-gravity distance between the case where the phase detection light flux is limited and the case where the phase detection light flux is not limited is obtained. Can be done. Then, the defocus amount for calculating the defocus amount from the image shift amount is calculated by previously obtaining and storing the sensitivity distribution information of the focus detection pixels _SHA and _SHB and the aperture information of the imaging optical system by measurement or calculation. The focus conversion coefficient can be obtained.

The defocus amount 729 shown in FIG. 7A is defined as “DEF”, and the distance 730 from the position 703 (planned image plane position) of the main image sensor 102 to the exit pupil surface 701 is defined as “L”. The center-of-gravity spacing (baseline length (distance between points 725 and 726)) when the luminous flux for phase difference detection is limited is "G1", and the center-of-gravity spacing (baseline length (point)) when the luminous flux for phase difference detection is not limited. The distance between 727 and 728)) is defined as "G2". The image shift amount 731 is set to "PRED1", and the defocus conversion coefficient for converting the image shift amount 731 to the defocus amount is set to "K1". Further, the image shift amount 732 is set to "PRED2", and the defocus conversion coefficient for converting the image shift amount 732 to the defocus amount is set to "K2".

In this case, the defocus amount is obtained by "DEF = K1 × PRED1 = K2 × PRED2". Further, the defocus conversion coefficients K1 and K2 are obtained by "K1 = L / G1" and "K2 = L / G2", respectively. Since G1 <G2, K1> K2, which generally means that the value of the defocus conversion coefficient increases as the light flux used for focus detection is limited.

When the focus detection position is not near the optical axis, vignetting of the focal detection light flux occurs due to the exit pupil of the diaphragm located at a position other than the exit pupil surface 701. Further, due to the exit pupil corresponding to the lens holding frame other than the aperture of the imaging optical system, vignetting of the focal detection light flux occurs even when the F value is brighter than the exit pupil of the aperture.

Here, we have explained how the luminous flux is limited by the lens holding frame and the aperture. Not limited to this, the position where the luminous flux is limited also changes depending on the change in the distance (exit pupil distance) from the main image sensor 102 to the pupil formed by the optical image by each lens group of the zoom mechanism unit 1012 and the focus mechanism unit 1011. do. The baseline length changes depending on the position where the luminous flux is restricted, and the defocus conversion coefficient changes according to the conditions under which the luminous flux is restricted. That is, the sensitivity of the defocus amount to the image shift amount changes due to the limitation of the luminous flux. When the aperture setting value of the photographing lens at the time of focus detection becomes large, the value of the defocus conversion coefficient becomes large and the sensitivity of the defocus amount to the image shift amount becomes high. Depending on the conditions, focus detection by the main image pickup element 102 becomes impossible.

The image pickup device 1 performs focus detection by the sub-image sensor 203 when the accuracy is low or the focus detection cannot be performed by the focus detection using the output signal from the main image sensor 102. Therefore, next, a focus detection method in the sub-imaging optical system 20 will be described. FIG. 8 is a diagram illustrating the principle of triangulation using the sub-imaging optical system 20. FIG. 8A is a diagram geometrically showing the positional relationship between the sub-imaging optical system 20 and the subject S. FIG. 8B is a schematic diagram illustrating a method of obtaining the parallax Δ from the position of the subject image on the image areas 303 and 304.

When the subject S is present within the angle of view of the sub-imaging optical system 20, the optical image (subject image) of the subject S is formed in the image region 303 of the sub-imaging element 203 via the sub-optical lens barrel 201 and is subordinated. An image is formed on the image region 304 of the sub-imaging element 203 via the optical lens barrel 202. Assuming that any point of the subject image formed in the image area 303 is sa and the point corresponding to the point sa in the subject image formed in the image area 304 is sb, the pixel position when the point sa is mapped to the image area 303. And the pixel position of the point sb in the image region 304 are separated by the parallax Δ.

The focal length of the sub-optical barrels 201 and 202 is "f", the distance from the sub-imaging optical system 20 (the principal point of the sub-optical barrels 201 and 202) to the subject S is "L", and the sub-optical barrels 201 and 202. Let the distance between the optical axes of be "D". Assuming that "L" is a value sufficiently larger than "f", the relationship of L = D × f / Δ ... (Equation 1) holds. The parallax Δ can be obtained by performing the known phase difference processing in the sub-phase difference processing unit 205 from the images obtained from the image regions 303 and 304. Then, since "D" and "f" are known, the distance L is calculated by applying the obtained parallax Δ, "D" and "f" to the above equation 1, that is, the distance measurement is performed. It can be carried out.

In the distance measurement by the sub-imaging optical system 20, since the baseline length is defined by the mechanical configuration of the sub-optical lens barrels 201 and 202, the baseline length does not change, so that the distance measurement accuracy is improved by changing the optical conditions. Deterioration of reliability is unlikely to occur. Further, the optical condition settings such as the aperture setting in the sub-optical lens barrels 201 and 202 can be set to the conditions suitable for distance measurement.

Next, the image pickup operation in the image pickup apparatus 1 will be described. FIG. 9 is a flowchart illustrating an image pickup operation in the image pickup apparatus 1. Each process (step) indicated by the S number in the flowchart of FIG. 9 is realized by the control unit 14 executing a predetermined program to comprehensively control the operation of each unit constituting the image pickup apparatus 1.

The imaging operation is started by the operation of the operation unit 16 by the user or the control command of the control unit 14. In S501, the control unit 14 starts the operation of the main imaging optical system 10. As a result, the main optical lens barrel 101 is set to the initial operation state, and the image pickup acquisition operation by the main image sensor 102 is started. In S502, the control unit 14 determines the shooting conditions of the main imaging optical system 10 based on the instruction from the operation unit 16, the image signal obtained in S501, the AE calculation result in the main imaging signal processing unit 11, and the like. When the control unit 14 determines the shooting conditions, the control unit 14 controls the drive of the zoom mechanism unit 1012 and the aperture mechanism unit 1013 in the main optical lens barrel 101, and adjusts the sensitivity of the main image sensor 102 by exposure time and signal amplification. Etc. to control the exposure.

In S503, the control unit 14 determines whether or not to perform imaging (image storage) based on the instruction from the operation unit 16. When the control unit 14 determines that shooting is to be performed (YES in S503), the process proceeds to S504, and when it is determined that shooting is not performed (NO in S503), the process proceeds to S506. In S504, the control unit 14 performs a focus detection operation on the subject. In S504, the focus adjustment operation by the main imaging optical system 10 or the focus detection operation by the distance measuring operation in the sub-imaging optical system 20 is performed, and the details thereof will be described later with reference to the flowchart of FIG. In the first embodiment, the focus detection operation is performed when shooting (image storage), but the focus detection operation may be performed at all times.

In S505, the control unit 14 takes an image by the main imaging optical system 10 based on the focus detection result acquired in S504, and stores the image data generated through predetermined image processing in the image storage unit 18. In S506, the control unit 14 determines whether or not the instruction to end shooting has been received through the operation unit 16. When the control unit 14 determines that the instruction to end the shooting is not accepted (NO in S506), the process returns to S502, and when it is determined that the instruction to end the shooting is accepted (YES in S506), the control unit 14 waits for the image pickup device 1. As a state, it waits for an instruction from the user through the operation unit 16.

FIG. 10 is a flowchart of the focus detection operation in S504. When the focus detection operation is started, in S601, the control unit 14 drives the main image sensor 102 to acquire an image signal. This image signal includes both a signal output from the image pickup pixel and a signal output from the focus detection pixel. In S602, the control unit 14 acquires the aperture set value F of the aperture mechanism unit 1013 in the primary optical lens barrel 101. In S603, the control unit 14 determines whether or not the aperture set value F acquired in S602 is equal to or greater than a predetermined aperture threshold value Fth. This is because, as described with reference to FIG. 7, when the aperture diameter is made small (that is, the F value is made large), vignetting of the luminous flux occurs, and the baseline length of the focus detection pixel in the main image sensor 102 becomes short. This is because the sensitivity of the defocus amount becomes high. Therefore, it is determined whether or not the aperture set value F is equal to or greater than the aperture threshold value Fth as the lower limit value at which the reliability of the defocus amount due to the focus detection pixels of the main image sensor 102 cannot be maintained. When the control unit 14 determines that the aperture set value F is smaller than the aperture threshold value Fth (NO in S603), the process proceeds to S604, and when it is determined that the aperture set value F is equal to or greater than the aperture threshold value Fth (YES in S603). ), The process proceeds to S605.

In S604, the control unit 14 operates the main phase difference processing unit 12 to perform known phase difference processing using the signal output from the focus detection pixel among the image signals acquired in S601 to obtain a parallax signal. After S604, the control unit 14 advances the process to S610.

On the other hand, in S605, the control unit 14 determines that focus detection through the main image pickup optical system 10 is difficult, and determines whether or not the sub image pickup optical system 20 is operating in order to perform focus detection using the sub image pickup optical system 20. judge. In the image pickup apparatus 1, when it is not necessary to use the sub-imaging optical system 20 in order to suppress power consumption, the sub-imaging optical system 20 is put into a stopped state (sleep state). The time when it is not necessary to use the sub-imaging optical system 20 is when the state of the main imaging optical system 10 is set under the condition that the distance measurement by the main imaging optical system 10 is not hindered. Therefore, the determination of S605 is required. However, the image pickup apparatus 1 may be configured to be constantly maintained in an operating state even if the sub-imaging optical system 20 is not actually used, and in this case, the determination of S605 is unnecessary. After the determination of S603 becomes NO, the process proceeds to S607. Whether or not the sub-imaging optical system 20 is always in the operating state depends on the default setting or the user setting. When the control unit 14 determines that the sub-imaging optical system 20 is in operation (YES in S605), the process proceeds to S607, and when it is determined that the sub-imaging optical system 20 is not in operation (NO in S605). , The process proceeds to S606.

In S606, the control unit 14 starts driving the sub-imaging optical system 20, and then advances the processing to S607. In S607, the control unit 14 sets the imaging conditions in the sub-imaging optical system 20 based on the image signal obtained from the sub-imaging element 203. In the subsequent S608, the control unit 14 performs image pickup by the sub-image sensor 203 and acquires images of the image areas 303 and 304 described with reference to FIG. In S609, the control unit 14 operates the sub-phase difference processing unit 205 to perform known phase difference processing using the images of the image regions 303 and 304 acquired in S608 to obtain a parallax signal. After S609, the control unit 14 advances the process to S610.

In S610, the control unit 14 calculates the defocus amount based on the parallax signal obtained by the processing in S604 or S609, and determines whether the main imaging optical system 10 is in the focused state. When the control unit 14 determines that the subject is in focus (YES in S610), the process proceeds to S611, and when it is determined that the subject is not in focus (NO in S610), the control unit 14 advances the process to S612. In S611, the control unit 14 stops driving (as a sleep state) when the sub-imaging optical system 20 is being driven, and ends this process. As a result, the process can proceed to S505. On the other hand, in S612, the control unit 14 drives the focus mechanism unit 1011 in the main optical lens barrel 101, and returns the process to S601 in order to perform the focus detection operation again.

As described above, in the first embodiment, when the distance measuring performance of the main image sensor 102 by the image pickup surface phase difference method is deteriorated due to the control state of the main optical lens barrel 101, the sub image pickup optical system 20 Switch to the focus detection operation using the distance measurement operation by multiple optical systems. As a result, it is possible to realize an image pickup device capable of performing appropriate focus detection while applying the optimum image pickup conditions to the main image pickup optical system 10 in order to obtain a photographed image.

In the above description, as an example in which the performance in distance measurement using the image signal output from the main image sensor 102 is deteriorated due to the control state of the main optical lens barrel 101, the aperture in the main optical lens barrel 101 is reduced. The case where vignetting of the light beam occurs according to the setting condition of the mechanism unit 1013 has been taken up. Not limited to this, the optical system for distance measurement may be switched according to the change in the zoom position by the zoom mechanism unit 1012 and the change in the exit pupil distance due to the movement of the focus lens by the focus mechanism unit 1011. In that case, when the exit pupil distance is smaller than a predetermined threshold, distance measurement is performed using the main imaging optical system 10, and when the exit pupil distance is equal to or more than a predetermined threshold, distance measurement is performed using the sub-imaging optical system 20. You can do it.

<Second Embodiment>
In the second embodiment, the image pickup device that switches between the ranging operation using the main imaging optical system 10 and the ranging operation using the sub-imaging optical system 20 according to the noise state of the output signal from the main imaging element 102. explain. The image pickup apparatus according to the second embodiment and the image pickup apparatus 1 described in the first embodiment differ only in the content of the control executed by the control unit 14, and the other configurations are the same. Therefore, hereinafter, only the characteristic matters in the second embodiment will be described.

As described with reference to FIG. 5, the focus detection pixels SHA and SHB included in the main image sensor 102 are the exit pupils EPHA and EPHB through the openings _OPHA and _OPHB having a small opening area formed in the wiring layer CL, respectively. Receives the luminous flux from. Since the focus detection pixels _SHA and _SHB have a smaller aperture ratio than the image pickup pixel, the light receiving sensitivity is lower than that of the image pickup pixel. Therefore, the signal noise ratio (SNR) in the image signal output from the focus detection pixels _SHA and _SHB . ) Decreases. That is, the noise increases and the quality of the image signal deteriorates. In particular, in a low-light environment, the main image sensor 102 raises the SNR by a signal amplification operation, but at this time, noise is also amplified in the image signals output from the focus detection pixels _SHA and _SHB having low light receiving sensitivity. As a result, the signal variation becomes large, and the reliability of the distance measurement accuracy is lowered.

In order to deal with this problem, in the second embodiment, the sub-imaging optical system 20 is driven according to the SNR of the main image sensor 102 to perform distance measurement. This is because the baseline length of the sub-imaging optical system 20 is defined by the mechanical configuration of the sub-optical lens barrels 201 and 202, so that the focal detection by the sub-imaging optical system 20 is better than the focal detection by the main imaging optical system 10. This is because the influence of noise superimposed on the image signal is small.

FIG. 11 is a flowchart of the focus detection operation according to the second embodiment. The series of processes shown in the flowchart of FIG. 11 is a process that can be executed in S504 of the flowchart of FIG. 9, and is a process that is executed in place of the series of processes shown in the flowchart of FIG. Of the various processes shown in the flowchart of FIG. 11, the same processes as those shown in the flowchart of FIG. 10 will be described to that effect and duplicated explanations will be omitted.

Since the process of S701 is the same as the process of S601, the description thereof will be omitted. In S702, the control unit 14 acquires the SNR of the main image sensor 102. Specifically, by obtaining the output standard deviation of the light-shielding pixel OPB (not shown) provided in the main image sensor 102 in order to detect the black level of the image sensor from the image signal obtained in S701, the main image sensor Obtain the SNR of 102.

In S703, the control unit 14 determines whether or not the SNR of the main image sensor 102 acquired in S702 is equal to or higher than a predetermined SNR threshold value SNRth. Specifically, whether the SNR of the main image sensor 102 is equal to or higher than the SNR threshold _SNRth , which is the minimum value that can ensure the reliability of the _defocus amount by the focus detection pixels SHA and SHB in the main image sensor 102. judge. When the control unit 14 determines that the SNR of the main image sensor 102 is equal to or higher than the SNR threshold _SNRth (YES in S703), the control unit 14 proceeds to S704 and _performs focus detection by the focus detection pixels SHA and SHB of the main image sensor 102. Perform processing. On the other hand, when the control unit 14 determines that the SNR of the main image sensor 102 is smaller than the SNR threshold value SNRth (NO in S703), the processing proceeds to S705, and the distance measuring operation using the sub-imaging optical system 20 is performed. Since the processing of S704 to S712 is the same as the processing of S604 to S612, the description thereof will be omitted.

As described above, in the second embodiment, when the SNR of the image signal of the main image sensor 102 is small (there is a lot of noise) and the reliability of the focus detection by the focus detection pixels _SHA and _SHB is lowered, the mechanical configuration is used. Distance measurement and focus detection are performed using the sub-imaging optical system 20 having a defined baseline length. This makes it possible to perform an appropriate focus detection operation.

In the second embodiment, a configuration for switching the distance measuring means according to the SNR (noise magnitude) of the image signal has been described. However, the present invention is not limited to this, the value of the gain applied to the main image sensor 102, the standard deviation of the output signal from the image pickup pixel within the range measurement range of the main image sensor 102, and the signal amplitude ratios of the focus detection pixels _SHA and _SHB . The distance measuring means may be switched according to any of the above. When the value of the gain applied to the main image sensor 102 is smaller than a predetermined threshold value, distance measurement is performed using the main image pickup optical system 10, and when the value of the gain applied to the main image sensor 102 is equal to or greater than the same threshold value, sub-imaging is performed. Distance measurement is performed using the optical system 20. When the standard deviation of the output signal from the image pickup pixel within the distance measurement range of the main image pickup element 102 is smaller than a predetermined threshold value, distance measurement is performed using the main image pickup optical system 10, and the standard deviation is equal to or more than the same threshold value. In some cases, distance measurement is performed using the sub-imaging optical system 20. When the signal amplitude ratios of the focus detection pixels _SHA and _SHB are smaller than a predetermined threshold value, distance measurement is performed using the main imaging optical system 10, and when the signal amplitude ratio is equal to or higher than the same threshold value, the sub-imaging optical system is used. Distance measurement using the system 20 is performed.

<Third Embodiment>
In the third embodiment, an image pickup device that switches from a distance measurement operation using the main image pickup optical system 10 to a distance measurement operation using the sub-imaging optical system 20 according to the signal saturation state of the main image pickup element will be described. The structure of the main image sensor is different between the image pickup device according to the third embodiment and the image pickup device 1 described in the first embodiment, and the content of the control executed by the control unit 14 is different accordingly. , Other configurations are the same. Therefore, hereinafter, only the characteristic matters relating to the third embodiment will be described.

FIG. 12A is a diagram illustrating a pixel arrangement in the main image sensor 102A according to the third embodiment. FIG. 12B is a diagram illustrating the structure of the pixels 1201 constituting the main image sensor 102A. The main image sensor 102 in the first embodiment has a structure having an independent image pickup pixel and a focus detection pixel, but the main image sensor 102A in the third embodiment has a configuration in which the image pickup pixel also has a focus detection function. Become.

The pixel 1201 has a photoelectric conversion unit 1201L (first photoelectric conversion unit) that receives a part of the pupil of the optical system and a photoelectric conversion unit 1201R (second photoelectric conversion unit) that receives a part of the pupil different from the photoelectric conversion unit 1201L. It has a photoelectric conversion unit). Similarly, pixel 1202 has photoelectric conversion units 1202L, 1202R, pixel 1203 has photoelectric conversion units 1203L, 1203R, and pixel 1204 has photoelectric conversion units 1204L, 1204R. The description of other imaging pixels will be omitted. In order to provide a two-dimensional image, the pixel arrangement in the main image pickup element 102A is such that a plurality of image pickup pixels including the pixel 1201 are arranged in a two-dimensional array shape.

Since the structures of the plurality of image pickup pixels constituting the main image pickup element 102A are all the same, the structure thereof will be described here by taking the pixel 1201 as an example. Pixel 1201 has one microlens 121 and photodiodes 122,123 (hereinafter referred to as “PD122,123”) provided under the microlens 121. Further, the pixel 1201 includes transfer switches 124 and 125, which are transistors that read out the signals of the PDs 122 and 123, and a floating diffusion 126 (hereinafter referred to as “FD126”) that temporarily stores the signals of the PDs 122 and 123. Have. As described above, the pixel 1201 has the same configuration as the image pickup pixel in the main image pickup element 102 in the first embodiment except that the pupil regions having different optical systems are read out by the PDs 122 and 123.

FIG. 13 is a schematic view showing how the luminous flux emitted from the exit pupil of the photographing lens is incident on the main image sensor 102A. FIG. 13 shows three imaging pixels that are part of the cross section of row 1205 shown in FIG. 12 (a). One imaging pixel has a color filter 1302 in addition to one microlens 121 and PD122,123.

With respect to the image pickup pixel having the central microlens 121, the center of the light flux emitted from the exit pupil 1303 of the photographing lens is defined as the optical axis 1304. The light emitted from the exit pupil 1303 forms an image on the main image sensor 102A centering on the optical axis 1304. The light rays 1307 and 1308 represent the outermost light rays that pass through the pupil region 1305, which is a part of the exit pupil of the photographing lens. Further, the light rays 1309 and 1310 represent the outermost light rays passing through the pupil region 1306, which is a part of the exit pupil of the photographing lens. As can be seen from FIG. 13, of the luminous flux emitted from the exit pupil 1303, the upper luminous flux is incident on the PD 122 and the lower luminous flux is incident on the PD 123 with the optical axis 1304 as a boundary. That is, the PDs 122 and 123 receive light that has passed through different pupil regions 1305 and 1306 of the exit pupils 1303 of the photographing lens, respectively.

The pixels 1201, 1202, 1203, 1204 shown in FIG. 12A are obtained from PD1201L, 1202L, 1203L, 1204L, ... The image to be created is referred to as an A image. Then, the signal is called an A image signal. Further, the image obtained from the PD1201R, 1202R, 1203R, 1204R, ... Corresponding to the PD123 that receives the light flux emitted from the other exit pupil across the optical axis 1304 is referred to as a B image. Then, the signal is called a B image signal.

The light fluxes emitted from the pupil regions 1305 and 1306 obtained by equally dividing the exit pupil 1303 around the optical axis 1304 are received by the PDs 122 and 123. As a result, a change (phase difference) appears in the address interval of the pixel 1201 or the like in which the outputs of the A image signal and the B image signal corresponding to the pupil regions 1305 and 1306 appear according to the change in the focal state. Therefore, the defocus amount can be calculated by detecting this address interval (detecting the phase difference).

FIG. 14 is an equivalent circuit diagram illustrating the pixel circuit configuration of the main image sensor 102A, and here, one pixel 1201 is schematically shown. The transfer switches 124 and 125 of the pixel 1201 are driven by the transfer pulses φTx1 and φTx2, respectively, and transfer the charges generated by the corresponding PDs 122 and 123 to the FD 126. The FD 126 serves as a buffer for temporarily storing electric charges. A floating diffusion amplifier is composed of an FD 126, an amplification MOS amplifier 141 that functions as a source follower, and a constant current source 145 connected to a vertical output line 143. The signal charge of the FD126 of the pixel selected by the selection switch 142 that selects the pixel by the vertical selection pulse φSEL is converted into a voltage by the floating diffusion amplifier, output to the vertical output line 143, and read out through a horizontal transfer signal line (not shown). .. The FD 126 is reset by VDD when the reset switch 144 is turned on in response to the reset pulse φRES.

The PDs 122 and 123 have corresponding transfer switches 124 and 125, respectively, but share a circuit used for signal reading after the FD126, whereby the pixels can be reduced. Further, the wiring that provides the transfer pulses φTx1 and φTx2 is shared by the pixels arranged in each row.

FIG. 15 is a timing chart for explaining the driving method (driving pattern) of the main image pickup device 102, and shows the relationship between the phase difference information and the operation of reading the image signal. In the period t151, the reset pulse φRES and the transfer pulses φTx1 and φTx2 are simultaneously controlled to the high potential H. As a result, the reset switch 144 and the transfer switches 124, 125 are turned on, and the potentials of the PD 122, 123 and the FD 126 are reset to the initial potential VDD. After that, when the transfer pulses φTx1 and φTx2 reach the low potential L, charge accumulation in the PDs 122 and 123 is started.

After a predetermined predetermined time has elapsed from the start of charge accumulation, the vertical selection pulse φSEL is set to the high potential H in the period t153, and the selection switch 142 is turned on to select the read line and read the signal for one line. The operation starts. At the same time, the reset pulse φRES is set to the low potential L, and the reset of the FD 126 is released. In the period t154, the φTN is set to the high potential H, so that the N signal, which is the reset signal of the FD 126, is read out and stored. Although not shown, in the main image sensor 102, the potential of the FD 126 is read out via the vertical output line 143 based on the control of φTN, φS1 and φS2, and each signal is stored.

Next, in the period t155, the transfer pulses φTx1 and φS1 are simultaneously set to the high potential H, and the transfer switch 124 is turned on, so that the first PD signal, which is an addition signal of the optical signal (image signal) and the noise signal of the PD 122, is read. It is taken out and stored in the main image sensor 102. After that, without turning on the reset switch 144, the transfer pulses φTx1, φTx2 and φS2 are simultaneously set to the high potential H during the period t156, so that the transfer switches 124 and 125 are turned on. As a result, the second PD signal, which is an addition signal obtained by mixing the optical signals and noise signals of the PDs 122 and 123, is read out and stored in the main image sensor 102.

Since the transfer pulse φTx1 is turned on in the period t155 and the first PD signal is read out from the PD 122 to the FD 126, the transfer pulse φTx1 may be in the off state in the period t156. Strictly speaking, the accumulation period t152 is from the end of the period t151 to the end of the period t156. The timing for resetting the PD 123 may be delayed by the time difference between the period t155 and the period t156 with the transfer pulse φTx2 as the high potential H.

The main image pickup element 102A is an A image signal obtained by subtracting a noise signal from the first PD signal from the N signal, the first PD signal, and the second PD signal read by the above operation, and an image signal obtained by differentizing the noise signal from the second PD signal. Is output from. Since the image signal obtained by subtracting the noise signal from the second PD signal is a signal obtained by synthesizing the signals of PD 122 and 123, the B image signal can be generated by subtracting the A image signal from this image signal. The main phase difference processing unit 12 generates a phase difference signal by using the A image signal and the B image signal thus obtained.

Since the main image sensor 102A acquires a pupil-divided signal from a plurality of (two) photodiodes constituting each pixel, depending on the structure of the photodiode, the signal of one photodiode becomes the other photodiode at high brightness. Phenomena such as leakage may occur. In this case, it may not be possible to accurately read the signal due to pupil division. That is, in a shooting environment in which the brightness of the subject is at a certain level or higher, the reliability of the signal acquired from the main image sensor 102A may decrease, and the calculation accuracy of focus detection may decrease.

In order to deal with this problem, in the third embodiment, the A image signal or the image signal (addition signal of the A image signal and the B image signal) of the pixels in the ranging area arbitrarily set in the plane of the main image sensor 102A. ) Counts the number of pixels whose output value is above a certain level. Then, when the ratio of the counted number of pixels to the total number of pixels in the distance measuring area becomes equal to or more than a predetermined threshold value, the distance measurement by the main imaging optical system 10 is switched to the distance measurement by the sub-imaging optical system 20. And perform focus detection.

FIG. 16 is a flowchart of the focus detection operation in the image pickup apparatus according to the third embodiment. The image pickup operation of the image pickup apparatus according to the third embodiment is performed in the same manner as the image pickup operation of the image pickup apparatus 1 shown in FIG. Therefore, the series of processes shown in the flowchart of FIG. 16 is a process that can be executed in S504 of the flowchart of FIG. 9, and is a process that is executed in place of the series of processes shown in the flowchart of FIG. Of the various processes shown in the flowchart of FIG. 16, the same processes as those shown in the flowchart of FIG. 10 will be described to that effect and duplicated explanations will be omitted.

When the focus detection operation is started, in S801, the control unit 14 drives the main image sensor 102 to perform the operation described with reference to FIG. 15 to acquire an image signal. In S802, the control unit 14 calculates the number of image pickup pixels NP in the ranging area set in the plane of the main image pickup element 102. To calculate the number of imaged pixels NP, the count end condition for the saturated pixel counting operation in the ranging area to be performed later is determined, and the saturated pixel ratio of the A image pixel and the B image pixel in the ranging area is obtained. Because. In S803, the control unit 14 sets the value of the A image saturation count CA, which is counted when the output level of the A image signal output from the pixels in the ranging area exceeds a certain threshold value, to zero (0). Perform the conversion process. Further, in S803, the control unit 14 counts the image saturation count when the output level of the image signal (addition signal of the A image signal and the B image signal) output from the pixels in the ranging area exceeds a certain threshold value. Initialization processing is performed to set the CI value to (0).

In S804, the control unit 14 sets “n”, which indicates a pixel address in the ranging area, to zero (0) as an initial value. Note that "n" is an integer value of 0 or more. Further, the pixel address in the range-finding area is defined as, for example, a reference point such as the upper left corner in the rectangular range-finding area as pixel address 0, and all the pixel addresses in the range-finding area are individually indicated. be. In S805, the control unit 14 detects the A image output value SA (n) of the pixel address n in the ranging area of the main image sensor 102A from the image signal obtained in S801.

In S806, the control unit 14 determines whether or not the A image output value SA (n) detected in S805 is equal to or higher than the predetermined A image output threshold value SAth. When the control unit 14 determines that the A image output value SA (n) is equal to or higher than the A image output threshold value SAth (YES in S806), the process proceeds to S807, and the A image output value SA (n) outputs the A image. If it is determined that it is smaller than the threshold value SAth (NO in S806), the process proceeds to S808. In S807, the control unit 14 increments the value of the A image saturation count CA by 1, and then advances the processing to S808.

Here, the reason for setting the threshold value for the image signal at the time of executing the focus detection operation will be described. FIG. 17A is a schematic diagram illustrating charge leakage between PDs 122 and 123, and schematically shows the state of potential energy of PDs 122 and 123. Here, it is assumed that the amount of incident light on the PD 122 is large and the amount of incident light on the PD 123 is small. When the amount of charge generated and accumulated in PD 122 exceeds a certain level, the charge generated in PD 122 exceeds the potential barrier between PD 122 and 123 and leaks to PD 123.

FIG. 17B is a diagram showing the relationship between the amount of light incident on the main image sensor 102 and the output signal. When the electric charge accumulated in the PD 122 increases, leakage from the PD 122 to the PD 123 occurs. That is, a part of the signal of PD122 starts to be mixed with the signal of PD123. The A image output threshold value SAth is set to a level at which mixing begins and signal reliability begins to be impaired. Here, a case where the amount of incident light on the PD 122 is relatively larger than the amount of incident light on the PD 123 has been described. On the contrary, when the amount of incident light on the PD 123 is relatively larger than the amount of incident light on the PD 122, the image output threshold SIth is set for the image signal SI. The image signal SI is an addition signal of an A image signal and a B image signal.

The A image output threshold value SAth and the image output threshold value SIth are set according to the image signal level in the level determination process in the main phase difference processing unit 12. Therefore, when the gain setting value of the main image sensor 102 is changed by the ISO sensitivity setting of the image pickup device or the like, the control unit 14 changes the values of the A image output threshold value SAth and the image output threshold value SIth.

The above description with respect to FIGS. 17A and 17B is an explanation when the ISO sensitivity setting is low. On the other hand, the relationship between the A image output threshold value SAth and the image output threshold value SIth when the ISO sensitivity setting is high and the gain setting value of the main image sensor 102 is large is as shown in FIG. 17 (c). That is, when the gain setting value of the main image sensor 102A is large, a small signal corresponding to a small amount of incident light is amplified and used. In this case, signal mixing does not occur between the pixels as described with reference to FIG. 17 (a).

However, in the case of a signal level at which the image signal SI is saturated, the main phase difference processing unit 12 cannot correctly generate a B image signal. Specifically, the graph line shown by the broken line in FIG. 17C shows the change in the B image signal when the A image signal is subtracted from the saturated level image signal SI to calculate the B image signal. The level of the B image signal increases as the amount of incident light increases. On the other hand, the graph line shown by the alternate long and short dash line in FIG. 17C shows the level change of the B image signal when the image signal SI is at the saturation level. When the image signal SI is saturated, the level of the A image signal is subtracted from the saturation level (constant level), so that the level of the B image signal decreases as the amount of incident light increases. As a result, the reliability of the B image signal is lowered. In order to avoid the occurrence of such a problem, the control unit 14 secures the reliability of the signal by setting the A image output threshold value SAth and the image output threshold value SIth to values different from those at the time of setting the low ISO sensitivity. Is possible.

Returning to the description of the flowchart of FIG. In S808, the control unit 14 detects the image signal output value SI (n) of the pixel address n in the ranging area of the main image sensor 102A. In S809, the control unit 14 determines whether or not the image signal output value SI (n) detected in S808 is equal to or greater than the image output threshold value SIth described with reference to FIG. When the control unit 14 determines that the image signal output value SI (n) is equal to or higher than the image output threshold value SIth (YES in S809), the processing proceeds to S810, and the image signal output value SI (n) is the image output threshold value SIth. If it is determined to be smaller (NO in S809), the process proceeds to S811.

In S810, the control unit 14 increments the value of the image saturation count CI by 1, and then advances the processing to S811. In S811, the control unit 14 increments the pixel address n by 1 in order to check the image signal level of the next pixel address (confirm the pixel next to the pixel address n). In S812, the control unit 14 compares the number of imaged pixels NP with the pixel address n, and whether or not the saturation count operation (S805 to S810) for all the pixels in the ranging area is completed (NP> n holds true). Whether or not) is judged. When the control unit 14 determines that NP <n (YES in S812), the process proceeds to S813, and when it is determined that NP <n is not (n ≦ NP) (NO in S812), the control unit 14 performs the process in S805. Return to.

In S813, the control unit 14 calculates the saturation pixel ratio in the ranging area by using the latest A image saturation count CA and the image saturation count CI. Specifically, the A image saturation count ratio CRA is obtained by "CRA = CA / NP", and the image saturation count ratio CRI is obtained by "CRI = CI / NP". In S814, the control unit 14 determines whether or not the A image saturation count ratio CRA or the image saturation count ratio CRI calculated in S813 is equal to or greater than a predetermined saturation count ratio threshold value Cth. The saturation count ratio threshold value Cth represents the upper limit of the range in which the reliability of the defocus amount obtained by the main image sensor 102A can be maintained (the lower limit of the range in which the reliability of the defocus amount decreases).

When the A image saturation count ratio CRA or the image saturation count ratio CRI is smaller than the saturation count ratio threshold value Cth, it is possible to perform distance measurement using the main imaging optical system 10. Therefore, if the determination in S814 is NO, the control unit 14 advances the process to S815. In S815, the control unit 14 operates the main parallax processing unit 12 to perform known parallax processing using the A image signal and the B image signal obtained by the imaging operation described with reference to FIG. 15. Obtain a parallax signal. After S815, the control unit 14 advances the process to S821.

When the A image saturation count ratio CRA or the image saturation count ratio CRI is equal to or higher than the saturation count ratio threshold value Cth, the focus detection accuracy in the main imaging optical system 10 is lowered. Therefore, when the determination in S814 is YES, the control unit 14 advances the process to S816 in order to switch from the distance measurement using the main imaging optical system 10 to the distance measurement using the sub-imaging optical system 20. .. The processing of S816 to S823 is the same as the processing of S604 to S612 (see FIG. 10) except that the processing is returned to S801 after S822, and thus the description thereof will be omitted.

As described above, in the third embodiment, in the main image pickup element 102A having a pixel structure having both focus detection and image pickup, when the reliability of distance measurement using the main image pickup optical system 10 cannot be guaranteed, the sub-imaging optics Switch to distance measurement using system 20. This makes it possible to perform appropriate distance measurement while applying the optimum shooting conditions to the main imaging optical system 10.

Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various embodiments within the range not deviating from the gist of the present invention are also included in the present invention. included. Further, each of the above-described embodiments is merely an embodiment of the present invention, and each embodiment can be appropriately combined.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1, 10A Main image pickup optical system 12 Main phase difference processing unit 14 Control unit 20, 20A Sub image pickup optical system 101 Main optical lens barrel 102, 102A Main image pickup element 201, 202 Sub optical lens barrel 203 Sub image pickup element 205 Sub-phase difference processing unit 1013 Aperture mechanism unit

Claims (20)

The first optical system, the first image pickup pixel in which the light beam passing through the exit pupil of the first optical system is imaged, and the second light beam passing through different pupil regions of the first optical system are imaged. A main image pickup element having an image pickup pixel of the above, and a main optical unit having the image pickup pixel of
It has a second optical system, a third optical system, and a sub-image pickup element, and the luminous flux that has passed through the second optical system and the light flux that has passed through the third optical system are the sub-image pickup elements, respectively. A sub-optical unit that forms an image in different regions and
A generation means for generating a phase difference signal using an output signal from the main image sensor or the sub image sensor, and
The primary optical unit and the sub-optical unit are controlled, and either the output signal from the main image sensor or the output signal from the sub-image sensor is selected according to the control state of the primary optical unit to the subject. Equipped with a control means for distance measurement
An image pickup apparatus in which the first optical system, the second optical system, and the third optical system are independent optical systems . The control state of the main optical unit is a state in which the aperture set value in the first optical system is controlled .
When the aperture set value is smaller than a predetermined threshold value, the control means measures a distance to the subject using an output signal from the second image pickup pixel of the main image sensor, and the aperture set value is the predetermined value. The image pickup apparatus according to claim 1, wherein when the value is equal to or more than the threshold value of, the distance is measured with respect to the subject using the output signal of the sub-image sensor. The control state of the main optical unit is a state in which the exit pupil distance in the first optical system is controlled .
When the exit pupil distance is smaller than a predetermined threshold value, the control means measures a distance to the subject using an output signal from the second image pickup pixel of the main image sensor, and the exit pupil distance is the predetermined value. The image pickup apparatus according to claim 1, wherein when the value is equal to or more than the threshold value of, the distance is measured with respect to the subject using the output signal from the sub-image sensor. The control state of the main optical unit is a state in which the gain applied to the main image sensor is controlled .
When the value of the gain applied to the main image sensor is smaller than a predetermined threshold value, the control means measures the distance to the subject using the output signal from the second image pickup pixel of the main image sensor, and obtains the gain. The image pickup apparatus according to claim 1, wherein when the value is equal to or more than the predetermined threshold value, the distance is measured with respect to the subject by using the output signal from the sub-image sensor. The first optical system, the first image pickup pixel in which the light beam passing through the exit pupil of the first optical system is imaged, and the second light beam passing through different pupil regions of the first optical system are imaged. A main image pickup element having an image pickup pixel of the above, and a main optical unit having the image pickup pixel of
It has a second optical system, a third optical system, and a sub-image pickup element, and the luminous flux that has passed through the second optical system and the light flux that has passed through the third optical system are the sub-image pickup elements, respectively. A sub-optical unit that forms an image in different regions and
A generation means for generating a phase difference signal using an output signal from the main image sensor or the sub image sensor, and
The main optical unit and the sub-optical unit are controlled, and either the output signal from the main image sensor or the output signal from the sub-image sensor is selected according to the image signal obtained from the main image sensor. It is equipped with a control means for measuring the distance to the subject .
An image pickup apparatus in which the first optical system, the second optical system, and the third optical system are independent optical systems . When the signal noise ratio of the image signal obtained from the main image sensor is smaller than a predetermined threshold value, the control means measures the distance using the output signal from the sub image sensor, and the signal noise ratio is the predetermined value. The image pickup apparatus according to claim 5, wherein when the value is equal to or greater than the threshold value of, the distance measurement is performed using the output signal from the main image sensor. The control means uses the output signal from the main image sensor to measure the distance when the standard deviation of the output signal from the first image pickup pixel within the distance measurement range of the main image sensor is smaller than a predetermined threshold value. The image pickup apparatus according to claim 5, wherein when the standard deviation is equal to or greater than the predetermined threshold value, distance measurement is performed using an output signal from the sub-image sensor. When the signal amplitude ratio of the output signal from the second image pickup pixel is smaller than a predetermined threshold value, the control means measures the distance using the output signal from the main image pickup element, and the signal amplitude ratio is the predetermined value. The image pickup apparatus according to claim 5, wherein when the value is equal to or more than the threshold value of, the distance is measured by using the output signal from the sub-image sensor. A first optical system and a main image sensor composed of pixels having a first photoelectric conversion unit and a second photoelectric conversion unit in which light flux passing through different pupil regions of the first optical system is imaged, respectively. With the main optical unit
It has a second optical system, a third optical system, and a sub-image pickup element, and the luminous flux that has passed through the second optical system and the light flux that has passed through the third optical system are the sub-image pickup elements, respectively. A sub-optical unit that forms an image in different regions and
A generation means for generating a phase difference signal using an output signal from the main image sensor or the sub image sensor, and
The main optical unit and the sub-optical unit are controlled, and either an output signal from the main image sensor or an output signal from the sub-image sensor is selected according to the output value of the signal obtained from the main image sensor. It is equipped with a control means for measuring the distance to the subject .
An image pickup apparatus in which the first optical system, the second optical system, and the third optical system are independent optical systems . The control means counts the number of pixels in the ranging area set in the main image sensor whose output value of the image signal from the first photoelectric conversion unit is at a predetermined level. When the ratio of the counted number of pixels to the total number of pixels in the distance measurement area is smaller than a predetermined threshold value, distance measurement is performed using the output signal from the main image sensor, and the ratio is the predetermined ratio. The image pickup apparatus according to claim 9, wherein when the value is equal to or more than a threshold value, distance measurement is performed using an output signal from the sub-image sensor. The control means outputs an additional signal of an image signal from the first photoelectric conversion unit and an image signal from the second photoelectric conversion unit from among the pixels in the ranging area set in the main image sensor. The number of pixels whose value is at a predetermined level is counted, and when the ratio of the counted number of pixels to the total number of pixels in the ranging area is smaller than the predetermined threshold value, the output from the main image sensor. The image pickup apparatus according to claim 9, wherein distance measurement is performed using a signal, and when the ratio is equal to or higher than the predetermined threshold value, distance measurement is performed using an output signal from the sub-image sensor. .. The image pickup apparatus according to claim 10, wherein the control means changes the threshold value according to the sensitivity of the main image pickup element. The image pickup apparatus according to any one of claims 1 to 12, wherein the shooting conditions of the sub-optical unit can be set to different conditions from the shooting conditions of the primary optical unit. One of claims 1 to 13, wherein the second optical system and the third optical system are provided apart from each other by a predetermined distance so that their optical axes are substantially parallel to each other. The imaging device according to item 1. The sub-image pickup element is one image pickup element in which a region in which a light flux passing through the second optical system is imaged and a region in which a light flux passing through the third optical system is imaged are different, or the second image pickup element. 1. The two image pickup elements are arranged in each of a region in which a light flux passing through the optical system of No. 1 is formed and a region in which a light flux passing through the third optical system is formed. 14. The image pickup apparatus according to any one of 14. When the generation means generates a phase difference signal using an output signal from the sub-imaging element, the image pickup signal of an image of a light beam that has passed through the second optical system and a light beam that has passed through the third optical system. The image pickup apparatus according to any one of claims 1 to 15, wherein a phase difference signal is generated from an image pickup signal of the image. It is a control method of the image pickup device.
A step of acquiring an optical image of light incident through a first optical system by a first image pickup device having an image pickup pixel and a focus detection pixel for performing distance measurement by an image pickup surface phase difference method.
A step of determining the reliability of the phase difference signal when a phase difference signal by the imaging surface phase difference method is generated by using the image signal of the optical image output from the focus detection pixel.
When it is determined that the reliability of the phase difference signal is high, a step of generating a phase difference signal by an imaging surface phase difference method using an image signal output from the focus detection pixel, and a step of generating the phase difference signal.
When it is determined that the reliability of the phase difference signal is low, the second optical system and the third optical system , which are independent of the first optical system and independent of each other, are used. A control method for an image pickup apparatus, comprising: a step of acquiring an optical image of a passing light beam by a second image pickup element and generating a phase difference signal from the image pickup signals of the two acquired optical images. When it is determined that the reliability of the phase difference signal is low, when the aperture setting value in the first optical system is equal to or higher than a predetermined value, the exit pupil distance in the first optical system is predetermined. When the value is equal to or greater than the value of, and when the magnitude of the gain applied to the first image sensor is equal to or greater than a predetermined value, the signal noise ratio of the image signal output from the first image sensor is greater than the predetermined value. The control method for an image pickup device according to claim 17, wherein the case is small. It is a control method of the image pickup device.
A step of acquiring an optical image of light incident through a first optical system by a first image sensor having pixels for performing image pickup and distance measurement by an image pickup surface phase difference method.
A step of determining the reliability of the phase difference signal when the phase difference signal by the image pickup surface phase difference method is generated by using the image signal output from the pixels in the ranging area set in the first image pickup element. When,
When it is determined that the reliability of the phase difference signal is high, a step of generating a phase difference signal by an imaging surface phase difference method using an image signal output from a pixel in the ranging area, and a step of generating the phase difference signal.
When it is determined that the reliability of the phase difference signal is low, the second optical system and the third optical system , which are independent of the first optical system and independent of each other, are used. A control method for an image pickup apparatus, comprising: a step of acquiring an optical image of a passing light beam by a second image pickup element and generating a phase difference signal from the image pickup signals of the two acquired optical images. A program comprising making a computer function as each means of the image pickup apparatus according to any one of claims 1 to 16 .

Images