JP7631245B2 - Image capture device, image capture element, image capture device control method, and image capture element control method - Google Patents

Description

The present invention relates to an autofocus function.

Imaging devices have an autofocus function (hereinafter, AF) that automatically adjusts the focus on a subject. It is desirable to complete the AF operation quickly so as not to miss a photographing opportunity even for a fast-moving subject. The AF operation is performed by moving the focus lens in the optical axis direction of the imaging optical system. When performing AF, first, all pixels of the image captured by the imaging element are read, and then the phase difference and contrast difference in the image are recognized in the image engine of the camera to obtain the defocus amount of the lens to the in-focus position. Then, the drive amount is calculated based on the obtained defocus amount and the current position information of the lens. After that, a drive command for moving the position of the focus lens is generated and transmitted to the lens side. The lens notifies the lens drive motor of the control information through the MPU. In this series of operations, the imaging element, the image engine, and the lens are controlled sequentially, so the AF operation takes a certain amount of time. In order to complete the AF operation quickly, Patent Document 1 discloses a technology in which the imaging element reads all pixels of the captured image, generates a lens drive command, and transmits it to the lens. By generating the lens drive commands without going through the image engine, AF operations can be completed faster than with conventional technology.

Incidentally, sensors equipped with AI have been developed in recent years. Patent Document 2 discloses an image sensor that has an image recognition function within the image sensor that receives light with a photodiode and reads out an image signal. The way in which the image signal is read can also be changed; for example, recognition can be performed with the image output by thinning out the image to 1/4 pixels, so that if recognition is possible with an image of 1/4 pixels, the image recognition result can be output before the completion of reading all pixels. For example, if recognition is not possible with 1/4 pixels, 2/4 pixels, or 3/4 pixels, the recognition result is output after reading all pixels (4/4 pixels). There is also technology that installs a scene recognition unit with a learning function within the image sensor. By controlling the driving of the image sensor based on the scene, it is possible to detect the subject without waiting for image processing on the image engine side.

JP 2017-98903 A Patent No. 6635221

In Patent Document 1, for example, when a subject targeted by a user moves at high speed into the frame of a fixed imaging device, the operation of setting an AF area for the subject may not be completed in time. In Patent Document 2, the current shooting scene is recognized in advance using a learning function installed in the imaging element, and the targeted subject itself can be detected at high speed, but there is no mention of technology for performing AF operation within the sensor. Furthermore, in both cases of Patent Document 1 and Patent Document 2, image correction such as distortion correction that was previously performed on the camera side cannot be performed, so if the subject is on the periphery, it may be difficult to detect the subject.

The purpose of the present invention is to enable high-speed, high-precision focus adjustment.

In order to solve the above problems, the imaging device of the present invention is an imaging device having an imaging element and an image processing device, and has a switching means for switching control information output for use in driving a focus lens to either first control information for controlling the driving of the focus lens generated by the imaging element or second control information for controlling the driving of the focus lens generated by the image processing device , the control information including information indicating the driving amount of the focus lens, the imaging element having a processor different from the image processing device, and generating the first control information using the result of subject detection inside the imaging device, without using the result of subject detection in the image processing device .

The present invention enables high-speed, high-precision focus adjustment.

FIG. 1 is a diagram illustrating a configuration of an imaging apparatus according to a first embodiment. FIG. 1 is a diagram illustrating image recognition. FIG. 13 is a diagram illustrating thinning readout. FIG. 1 is a diagram illustrating the timing of a conventional AF process. 5A to 5C are diagrams illustrating the timing of AF processing in the present embodiment. 5 is a flowchart showing a process relating to AF in the image sensor. 13 is a flowchart showing a process of a control signal switching unit. FIG. 13 is a diagram showing a configuration of an imaging apparatus according to a second embodiment. FIG. 13 is a diagram showing a configuration of an imaging apparatus according to a third embodiment.

First Embodiment
1 is a block diagram showing an example of the configuration of an imaging device 100. The imaging device 100 is, for example, an imaging device such as a digital single-lens reflex camera, a mirrorless camera, a video camera, or a compact digital camera. The imaging device 100 has a lens unit 101 and a camera unit 110. The imaging device 100 may be an imaging device in which the lens unit 101 and the camera unit 110 are integrated, or may be an interchangeable lens imaging device in which the lens unit 101 is detachable from the camera unit 110.

First, the configuration of the lens unit 101 will be described. The lens unit 101 forms an optical image of a subject on the image sensor 130. The lens unit 101 has an imaging optical system including multiple lenses such as a focus lens 102. The lens unit 101 further has a position detection unit 103, a drive circuit 104, a motor 105, a lens control unit 106, and an IF (interface) 107. The focus lens 102 is a lens whose focal length can be changed, and moves in the direction of the optical axis to adjust the focus.

The position detection unit 103 detects the position of the focus lens 102 and outputs it to the lens control unit 106. The drive circuit 104 and the motor 105 are drive units that drive the focus lens 102. The drive circuit 104 drives the motor 105 based on control information received from the lens control unit 106 for driving the focus lens 102 to a focus position. The motor 105 moves the focus lens 102 in the optical axis direction of the imaging optical system. The motor 105 is, for example, a stepping motor. Note that the motor 105 is not limited to a stepping motor, and may be an ultrasonic motor or the like.

The lens control unit 106 controls the overall operation of the lens unit 101. The lens control unit 106 is, for example, an MPU (Micro Processing Unit), and is connected to the camera unit 110 via an IF 107, which is a lens mount unit. The lens control unit 106 outputs control information received from the camera unit 110 via the IF 107 to the drive circuit 104 for driving the focus lens to a focused position. Based on the control information from the lens control unit 106, the drive circuit 104 and the motor 105 drive the focus lens 102, thereby performing focus control.

Next, the configuration of the camera unit 110 will be described. The camera unit 110 has an image processing engine 120 and an image sensor 130. In the camera unit 110 of this embodiment, the image processing engine 120 and the image sensor 130 can each generate control information such as a relative drive amount and an absolute drive amount related to driving to drive the focus lens 102 to a focus position. In the following, the drive information generated by the image sensor 130 is referred to as first control information, and the drive information generated by the image processing engine 120 is referred to as second drive information.

First, the image processing engine 120 will be described. The image processing engine 120 performs various image processes and a process of generating control information (second control information) for driving the focus lens 102 to a focus position. Furthermore, the image processing engine 120 performs a process of switching between the control information (first control information) generated by the image sensor 130 and the control information (second control information) generated by the image processing engine 120 as the control information used to drive the focus lens 102. The image processing engine 120 has an image processing unit 121, a recognition unit 122, a drive amount calculation unit 123, a control signal switching unit 124, and a control unit 125. The image processing engine 120 includes, for example, a CPU (Central Processing Unit) and a memory for storing a program. Each process by the image processing unit 121 to the control unit 125 of the image processing engine 120 is realized by the CPU executing a program stored in the memory.

The image processing unit 121 performs image processing such as image format conversion, resizing, color conversion, noise removal, distortion correction, chromatic aberration correction, and white balance adjustment on the image received from the imaging unit 131 of the image sensor 130. The image acquired by the image processing unit 121 from the image sensor 130 is an image acquired by reading all pixels. The recognition unit 122 calculates the defocus amount of the subject from the image processed by the image processing unit 121. The recognition unit 122 calculates the defocus amount of the subject, for example, by subject detection and phase difference AF or contrast AF processing. The drive amount calculation unit 123 calculates the drive amount (relative drive amount) of the focus lens 102 required for focusing based on the defocus amount calculated by the recognition unit 122. Note that the calculation method does not matter as long as the drive amount of the focus lens 102 required for focusing can be calculated. For example, the recognition unit 122 does not need to be used. In this embodiment, the drive amount calculation unit 123 sends the calculated drive amount of the focus lens 102 to the control signal switching unit 124.

The control signal switching unit 124 acquires the control information and priority information of the focus lens 102 calculated by the image sensor 130 and the drive information of the focus lens 102 calculated by the image processing engine 120. Then, the control signal switching unit 124 sends only the drive amount used to drive the focus lens 102 of the two drive amounts to the control unit 125. Which drive amount is used to control the drive of the focus lens 102 is determined by a drive amount calculation unit 133 in the image sensor 130 described later, and the control signal switching unit 124 switches the drive amount to be output according to the priority information determined by the drive amount calculation unit 133.

The control unit 125 transmits the control information acquired from the control signal switching unit 124 to the lens unit 101 via the driver 112 and the IF 111. In addition, when the control information acquired from the control signal switching unit 124 is a relative drive amount generated by the image processing engine 120, the control unit 125 updates the control information based on the position information of the focus lens 102. The control unit 125 calculates an absolute drive amount based on the relative drive amount and the position information of the focus lens 102 acquired from the lens unit 101, updates the control information, and outputs it to the lens unit 101. The control unit 125 calculates an absolute drive amount of the focus lens 102 based on the relative drive amount of the focus lens 102 and the position information of the focus lens 102 acquired from the lens unit 101. The control information of the focus lens 102 input to the lens unit 101 is information that defines the drive of the motor 105 required to move the focus lens 102 to a focus position. For example, when the motor 105 is a stepping motor, the control unit 125 generates information that defines the drive target pulse number as the control information. Also, if the control information is fixed byte length data, the control unit 125 transmits it to the driver 112 after the previous value transmission process is completed. In this way, the image processing engine 120 acquires an image of all pixels from the image sensor 130, and after image processing is performed by the image processing unit 121, the drive amount for AF is calculated. Note that, although an example in which the image processing engine 120 acquires all pixels and performs processing for AF control has been described in this embodiment, it is also possible to acquire and process image signals of more pixels than the pixels used by the image sensor 130 for AF control, even if not all pixels.

Next, the imaging element 130 will be described. The imaging element 130 is an image sensor that has a photoelectric conversion unit and photoelectrically converts an optical image formed through the imaging optical system of the lens unit 101 to output a signal related to the captured image. Furthermore, the imaging element 130 of this embodiment is equipped with AI and performs image recognition processing such as subject detection and scene recognition, and calculation processing of the drive amount of the focus lens 102 for focusing. The imaging element 130 has an imaging unit 131, a recognition unit 132, and a drive amount calculation unit 133. The imaging element 130 includes, for example, a CPU (Central Processing Unit) and a memory that stores a program. Each process by the imaging unit 131 to the determination unit 134 of the imaging element 130 is realized by the CPU executing a program stored in the memory.

The imaging unit 131 captures an image of a subject (optical image) formed via an imaging optical system by photoelectric conversion, and outputs an image signal. The recognition unit 132 is equipped with a learning function, and performs image recognition such as subject detection and scene recognition on the output of the imaging unit 131. The recognition unit 132 then calculates the defocus amount of the subject based on the results of the subject detection and scene recognition by image recognition. The recognition unit 132 sends the calculated defocus amount to the drive amount calculation unit 133. The recognition unit 132 also sends the results of image recognition such as subject detection and scene recognition to the determination unit 134.

The drive amount calculation unit 133 generates first control information for driving the focus lens 102 to a focus position. Specifically, the drive amount calculation unit 133 calculates the relative drive amount of the focus lens 102 required for focusing according to the defocus amount of the subject. The drive amount calculation unit 133 then calculates the absolute drive amount of the focus lens 102 based on the position information of the focus lens 102 acquired from the lens unit 101 and the calculated relative drive amount, and generates first drive information. The drive amount calculation unit 133 transmits the generated first drive information to the control signal switching unit 124 of the image processing engine 120 via IF 135 and IF 126.

The determination unit 134 determines whether the drive amount calculated by the image sensor 130 or the drive amount calculated by the image processing engine 120 should be prioritized, and generates priority information indicating the drive information to be prioritized. The determination unit 134 transmits the generated priority information to the control signal switching unit 124 of the image processing engine 120 via IF 135 and IF 126. The calculation of the defocus amount by the recognition unit 132 and the generation of the first control information by the drive amount calculation unit 133 and the generation of the priority information by the determination unit 134 are performed in parallel, and the first control information and the priority information are sent simultaneously from the image sensor 130 to the image processing engine 120. In this way, while the drive amount is calculated after image processing in the image processing engine 120, the image sensor 130 can calculate the control information without waiting for all pixels to be read out and without image processing, making it possible to calculate the control information faster.

The timing for generating control information for the focus lens 102 in the image sensor 130 will now be described with reference to Figs. 2 to 5. The image sensor 130 calculates a defocus amount in response to scene recognition and subject detection in the recognition unit 132, and calculates control information for the focus lens 102 based on the defocus amount. Fig. 2 is a diagram for explaining image recognition. Fig. 2 shows a case in which a baseball scene 140 is being photographed. When the recognition unit 132 of the image sensor 130 recognizes a baseball scene, for example, the recognition unit 132 calculates a defocus amount when it detects a white ball 141.

Furthermore, by performing scene recognition, it is possible to detect a specific object (for example, a white ball in baseball) according to the recognized scene before reading all pixels, and the calculation of control information for focusing on a specific object can be accelerated. FIG. 3 is a diagram explaining thinning readout. FIG. 3(A) shows an example of thinning out on a line-by-line basis. FIG. 3(B) shows an example of thinning out on a pixel-by-pixel basis. In FIGS. 3(A) and 3(B), the hatched areas indicate the pixels to be thinned out. Whether thinning out on a line-by-line basis as shown in FIG. 3(A) or thinning out the pixels to 1/4 as shown in FIG. 3(B), the white ball 141, which is the object to be detected in the baseball scene 140, can be detected.

FIG. 4 is a diagram for explaining the timing of the conventional AF process. When all pixels are read as shown in FIG. 2, as shown in FIG. 4, after the start of accumulation, it is necessary to wait for the completion of reading all pixels, and then start AF control to obtain the position of the lens, complement the amount of lens movement, and determine the final lens position. FIG. 5 is a diagram for explaining the timing of the AF process of this embodiment. When only some pixels are read as shown in FIG. 3, if the subject is recognized at the stage where, for example, 1/4 pixels are read as shown in FIG. 5, AF control can be started without waiting for the remaining 3/4 pixels to be read. In this way, control information for AF control can be generated at the timing when the image signals of some pixels are read, without synchronizing with the reading of the image signals. This allows the calculation of the final focus lens position to be completed earlier than in the past, regardless of whether phase difference AF or contrast AF is used. In this embodiment, the image processing engine 120 performs lens position interpolation (generation of first control information for the focus lens 102) after all pixels are read as shown in FIG. 4. On the other hand, the image sensor 130 performs lens position interpolation (generation of second control information for the focus lens 102) by reading some pixels without waiting for the reading of all pixels shown in FIG. 5. Therefore, the generation of the first control information by the image sensor 130 is faster than the generation of the second control information by the image processing engine 120.

In addition, if the location where the subject will appear can be predicted by image recognition such as recognition of the shooting scene (hereinafter referred to as scene recognition), the generation of the first control information in the image sensor 130 can be further accelerated by narrowing down the area for subject detection and focus detection in advance. For example, in Figures 2 and 3, the timing to focus is the moment of impact by the batter, and it can be predicted that the subject, the white ball 141, will appear at approximately waist height of the batter. If image plane phase difference AF is used, the line for image plane phase difference detection can be set only at the expected position of the subject's appearance, thereby speeding up the detection of the subject and the calculation of the defocus amount and the control information for the focus lens 102. Note that the detection of the subject and the calculation of the control information using the prediction of the subject position by image recognition may also be performed by the image processing engine 120.

Here, the determination by the determination unit 134 as to which of the first control information calculated in the image sensor 130 and the second control information calculated in the image processing engine 120 should be prioritized will be described. The calculation of the first control information of the focus lens 102 in the image sensor 130 can be performed at high speed because it can be performed without waiting for all pixels to be read out by image recognition. On the other hand, since image processing such as distortion aberration correction is not performed in the image sensor 130, the accuracy of scene recognition and subject detection may be lower than that of the image processing engine 120, and the accuracy of the control information and the AF based thereon may also be lower. The effect of not performing image processing such as distortion aberration correction in the image sensor 130 is particularly noticeable in the periphery of the screen where the image height is high, but on the other hand, the effect of not performing image processing is less likely to be seen near the center of the image height (center of the screen). The calculation of the second control information in the image processing engine 120 is performed after image processing by the image processing unit 121, so the accuracy of scene recognition and subject detection is high, and the accuracy of the calculation of the lens drive amount and the AF based thereon are also high. On the other hand, image processing is performed after all pixels are read out, and then subject detection and defocus amount calculation are performed, so the calculation of the drive amount in the image processing engine 120 is slower than the calculation of the drive amount in the image sensor 130.

Therefore, the determination unit 134 of the image sensor 130 determines which control information to prioritize based on the scene recognition detected by the recognition unit 132 and the position and movement (movement speed and movement direction) of the subject. When the reliability of the drive amount calculated in the image sensor 130 is high, the determination unit 134 notifies the control signal switching unit 124 of priority information indicating that the calculated value in the image sensor 130 is prioritized. For example, when the subject is in the center of the screen, the captured image is less susceptible to lens distortion and a good detection result can be obtained without going through the distortion correction calculation by the image processing unit 121, so the first control information calculated in the image sensor 130 is prioritized. When the subject's movement is random or when a stationary subject suddenly moves, and it is difficult to predict even by the predictive calculation process of the image processing engine 120 that predicts the focus position (when the reliability of the predictive calculation process is low), the first control information calculated in the image sensor 130 is prioritized. On the other hand, when the subject is in the periphery of the image, when the subject's moving speed is less than a predetermined value and moving in a fixed direction, etc., priority information indicating that the second control information calculated in the image processing engine 120 should be prioritized is notified to the control signal switching unit 124. In addition, in the case of a scene that requires high accuracy in subject detection, such as a dark scene, the second control information is prioritized. Note that the criteria for judgment are not limited to these, and are accumulated by the learning function of the AI of the image sensor 130. In addition, it is also possible to allow the user to set which drive amount is to be prioritized, and to determine the prioritized drive amount based on the user's setting. When the user sets the AF speed to be prioritized, the control information calculated in the image sensor 130 is prioritized.

The image sensor 130 transmits the first control information and priority information indicating which control information of the image sensor 130 or the image processing engine 120 should be prioritized to the control signal switching unit 124 in the image processing engine 120. The control signal switching unit 124 switches the control information to be sent to the control unit 125 based on the priority information acquired from the image sensor 130. When information that prioritizes the control information calculated by the image processing engine 120 is acquired, the control signal switching unit 124 sends the second control information calculated by the image processing engine 120 to the control unit 125. On the other hand, when information that prioritizes the drive amount calculated by the image sensor 130 is acquired, the control signal switching unit 124 sends the first control information calculated by the image sensor 130 to the control unit 125. The switching of the control information by the control signal switching unit 124 is performed without synchronizing with the image update period, i.e., the period of reading all pixels by the image sensor 130.

The control unit 125, which has acquired the second control information calculated by the image processing engine 120, calculates an absolute drive amount based on the lens position information and the relative drive amount, updates the second control information, and sends it to the driver 112. On the other hand, the control unit, which has acquired the first control information calculated by the image sensor 130, sends the control information as is to the driver 112. The control information sent to the driver 112 is sent to the lens control unit 106 via the IF 111 on the camera unit 110 side and the IF 107 on the lens unit 101 side. The control signal switching unit 124 prevents the control information on the image sensor 130 side and the control information on the image processing engine 120 side from being sent to the lens unit 101 in a mixed state.

The flow of processing related to AF in the image sensor 130 will be described. FIG. 6 is a flowchart showing the flow of processing in the image sensor 130 related to AF. In step S601, the image sensor 131 reads out image signals. In the image sensor 130, the processing of the next step S602 is performed when the reading of the thinned pixels is completed without waiting for the completion of the reading of all pixels. In steps S602 and S603, the recognition unit 132 performs image recognition. Specifically, in step S602, the recognition unit 132 performs scene recognition according to a learning model. If the scene is recognized, the subject position is predicted and the area in which the subject detection is performed is limited in step S603. In step S603, the recognition unit 132 performs subject detection. The scene recognition result, which is the result of image recognition, and the subject detection result are output to the drive amount calculation unit 133 and the determination unit 134.

In steps S604 to S606, the drive amount calculation unit 133 generates first control information. Specifically, in step S604, the drive amount calculation unit 133 acquires position information of the focus lens 102 from the lens unit 101. In step S605, the recognition unit 132 and the drive amount calculation unit 133 calculate a relative drive amount for driving the focus lens 102 to the in-focus position. First, the recognition unit 132 calculates a defocus amount, and the drive amount calculation unit 133 calculates a relative drive amount based on the defocus amount. Note that the calculation of the defocus amount by the recognition unit 132 may be performed in step S602, where image recognition processing is performed. In step S606, an absolute drive amount is calculated based on the position information of the focus lens 102 acquired in step S604 and step S603, and the first control information is generated.

In parallel with the processes of steps S604 to S606 for generating the first control information, the image sensor 130 performs a process for determining which control information is to be prioritized. In step S607, the determination unit 134 determines which of the first control information generated by the image sensor 130 or the second control information generated by the image processing engine 120 is to be prioritized, and generates priority information. The image sensor 130 then outputs the first control information generated in step S606 and the priority information generated in step S607 to the control signal switching unit 124 of the image processing engine 120.

Next, the flow of switching processing by the control signal switching unit 124 of the image processing engine 120 will be described. FIG. 7 is a flowchart showing processing by the control signal switching unit 124. In step S701, the control signal switching unit 124 determines whether to prioritize the first control information or the second control information based on the priority information acquired from the image sensor 130. If it is determined that the first control information generated by the image sensor 130 is to be prioritized, the process proceeds to step S702. On the other hand, if it is determined that the second control information generated by the image processing engine 120 is to be prioritized, the process proceeds to step S703.

In step S702, the control signal switching unit 124 sets the value for controlling the focus lens 102 to the in-focus position to be output to the lens unit 101 as the first control information generated by the image sensor 130. In step S703, the control signal switching unit 124 sets the value for controlling the focus lens 102 to the in-focus position to be output to the lens unit 101 as the second control information generated by the image processing engine 120. In step S704, the control signal switching unit 124 determines whether the output of the previous control information has been completed. If it is determined that the output of the previous control information has been completed, the process proceeds to step S705. On the other hand, if it is determined that the output of the previous control information has not been completed, the process of step S704 is repeated until it is completed. By the process of step S704, the control information can be switched after the transmission of the previous control information has been completed. In step S705, the control signal switching unit 124 outputs the control information determined in step S702 or step S703 to the driver 112. The control information output from the control signal switching unit 124 is input to the lens control unit 106 via the driver 112, the IF 111 on the camera unit 110 side, and the IF 102 on the lens unit 101 side.

As described above, according to this embodiment, it is possible to switch the control information for driving the focus lens to the in-focus position depending on the scene and subject, thereby achieving high-speed, high-precision AF operation.

Second Embodiment
In the first embodiment, an example has been described in which the absolute drive amount based on the position information of the focus lens 102 is calculated by both the image sensor 130 and the image processing engine 120. That is, the absolute drive amount corresponding to the first control information is calculated within the image sensor 130. In the second embodiment, an example will be described in which the image sensor 130 calculates even the relative drive amount for the first control information, and the image processing engine 120 calculates the absolute drive amount. Note that the calculation of the relative drive amount and absolute drive amount related to the second control information is similar to the first embodiment.

Below, only the differences from the first embodiment will be described. FIG. 8 is a diagram showing the configuration of the imaging device 100 of the second embodiment. The same components as those in the first embodiment are given the same reference numerals and their description will be omitted. In the first embodiment, the position information of the focus lens 102 acquired from the lens unit 101 is input to both the image sensor 130 and the image processing engine 120, but in the second embodiment, it is input only to the image processing engine 120. The camera-side IF 800 outputs the position information of the focus lens 102 acquired from the lens control unit 106 via the lens-side IF 107 only to the control unit 803 of the image processing engine 120, and does not output it to the image sensor 130. This makes it possible to reduce the physical wiring between the camera-side IF 800 and the image sensor 130.

The drive amount calculation unit 801 of the image sensor 130 does not acquire position information of the focus lens 102. Therefore, the drive amount calculation unit 801 calculates the relative drive amount for driving the focus lens 102 to the in-focus position based on the defocus amount calculated by the recognition unit 132, and generates first control information. The first control information, which is the relative drive amount generated by the drive amount calculation unit 801, is output to the control signal switching unit 802 of the image processing engine 120 together with the priority information determined by the determination unit 134.

The control signal switching unit 802 of the image processing engine 120 switches the control information to be output to the lens unit 101 based on the priority information. At this time, the control information output by the control signal switching unit 802 to the control unit 803 is a relative drive amount, whether it is the first control information or the second control information. The control unit 803 calculates the absolute drive amount of the focus lens 102 based on the relative drive amount acquired from the control signal switching unit 802 and the position information of the focus lens 102 acquired from the lens unit 101. Then, the control unit 803 outputs the calculated absolute drive amount to the lens unit 101 via the driver 112 and the IF 800.

In this way, in this embodiment, it is possible to switch control information for driving the focus lens to a focus position depending on the scene or subject, and to achieve high-speed, highly accurate AF operation. Furthermore, by not receiving position information of the focus lens 102 from the lens unit 101 in the image sensor 130, it is possible to reduce the amount of physical wiring between the lens unit 101 and the image sensor 130.

Third Embodiment
In the first and second embodiments, an example has been described in which the process of switching the control information of the focus lens 102 to be output to the lens unit 101 is performed within the image processing engine 120. In the third embodiment, an example will be described in which the process of switching the control information of the focus lens 102 to be output to the lens unit 101 is performed within the image sensor 130.

Below, only the differences from the first embodiment will be described. FIG. 9 is a diagram showing the configuration of the imaging device 100 of the third embodiment. The same components as those in the first embodiment are given the same reference numerals and their description will be omitted. The imaging element 930 has an imaging unit 131, a recognition unit 132, a drive amount calculation unit 901, a control signal switching unit 902, a control unit 903, an IF 135 with the image processing engine 120, and an IF 904 with the driver. The image processing engine 120 has an image processing unit 121, a recognition unit 122, and a drive amount calculation unit 905. Thus, in this embodiment, the imaging element 930 has a control signal switching unit 902 that switches the control information to be output to the lens unit 101.

In the first embodiment, the position information of the focus lens 102 acquired from the lens unit 101 is input to both the image sensor 130 and the image processing engine 120, but in the third embodiment, it is input only to the image sensor 930. The drive amount calculation unit 901 of the image sensor 930 calculates an absolute drive amount based on the relative drive amount calculated from the defocus amount and the position information of the focus lens 102 acquired from the lens unit 101, and generates first control information for driving the focus lens 102. The drive amount calculation unit 901 then outputs the generated first control information to the control signal switching unit 902 in the image sensor 930. The drive amount calculation unit 905 of the image processing engine 920 calculates a relative drive amount from the defocus amount, and generates second control information for driving the focus lens 102. The drive amount calculation unit 905 then outputs the generated second control information to the control signal switching unit 902 in the image sensor 930.

The control signal switching unit 902 acquires the first control information generated by the image sensor 930 from the drive amount calculation unit 901, and the second control information generated by the image processing engine 920 from the drive amount calculation unit 905 of the image processing engine 920. The control signal switching unit 902 decides which of the first control information generated by the image sensor 930 and the second control information generated by the image processing engine 920 is to be prioritized, and switches the control information to be output in accordance with the decision. In other words, the control signal switching unit 902 has the functions of the decision unit 134 and the control signal switching unit 124 of the first embodiment.

The control unit 903 outputs the control information input from the control signal switching unit 902 to the lens unit 101 via IF 904, driver 112, and IF 900. If the control information input from the control signal switching unit 902 is the first control information, the control unit 903 outputs the first control information to the driver 112 as is because the absolute drive amount has already been calculated. On the other hand, if the control information input from the control signal switching unit 902 is the second control information, the control unit 903 updates the second control information based on the position information of the focus lens 102 acquired from the lens unit 101. The control unit 903 calculates the absolute drive amount of the focus lens 102 based on the relative drive amount of the second control information and the position information of the focus lens 102, updates the second control information, and outputs it to the driver 112. In this embodiment, an example has been described in which the drive amount calculation unit 901 calculates the absolute drive amount related to the first control information, but the drive amount calculation unit 901 may only calculate the relative drive amount, and the control unit 903 may calculate the absolute drive amount related to the first control information.

As such, in this embodiment, compared to the first embodiment, the amount of physical wiring between the IF 900 and the image processing engine 920 can be reduced. On the other hand, an IF 904 is required in the image sensor 930 to connect to the driver 112.

In this way, in this embodiment, it is possible to switch control information for driving the focus lens to a focus position depending on the scene or subject, and to achieve high-speed, highly accurate AF operation. Furthermore, by not receiving position information of the focus lens 102 from the lens unit 101 in the image processing engine 920, it is possible to reduce the amount of physical wiring between the lens unit 101 and the image processing engine 920.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

REFERENCE SIGNS LIST 100 Imaging device 110 Camera unit 120 Image processing engine 121 Image processing unit 122 Recognition unit 123 Drive amount calculation unit 124 Control signal switching unit 125 Control unit 130 Imaging element 131 Imaging unit 132 Recognition unit 133 Drive amount calculation unit

Claims (16)

An imaging device having an imaging element and an image processing device,
a switching means for switching control information to be output for use in driving a focus lens between first control information for controlling the driving of the focus lens generated by the image sensor and second control information for controlling the driving of the focus lens generated by the image processing device;
the control information includes information indicating a drive amount of the focus lens,
The imaging device, characterized in that the imaging element has a processor different from the image processing device, and generates the first control information using a result of subject detection inside the imaging device, without using a result of subject detection in the image processing device. The imaging device according to claim 1, characterized in that the imaging element generates the first control information earlier than the image processing device generates the second control information. The imaging device according to claim 1 or 2, characterized in that the imaging element generates the first control information asynchronously with the readout of the image signal and outputs it to the switching means. The imaging device according to any one of claims 1 to 3, characterized in that the imaging element generates the first control information and outputs it to the switching means before image signals of all pixels are read out. The imaging device according to any one of claims 1 to 4, characterized in that the imaging element generates the first control information based on image signals of some pixels, and the image processing device generates the second control information based on image signals of all pixels that have been subjected to image processing. The imaging element includes:
an imaging means for capturing an image of a subject and outputting an image signal;
A recognition means for performing image recognition based on the image signal;
A generating unit that generates first control information for controlling driving of the focus lens;
a determination means for determining priority control information based on a result of the image recognition and generating priority information indicating the priority control information;
4. The imaging apparatus according to claim 1, wherein the switching unit switches the control information to be output based on the priority information. The imaging device according to claim 6, characterized in that the determining means determines the control information to be prioritized based on either the position and movement of the subject or the photographed scene. The imaging device according to claim 6 or 7, characterized in that the recognition means predicts the position of the subject by the image recognition, and detects the subject and calculates the defocus amount based on the predicted position of the subject. The imaging device according to any one of claims 1 to 8, characterized in that the switching means switches the control information asynchronously with a period in which the imaging element reads out an image signal. The imaging device according to any one of claims 1 to 9, characterized in that the switching means switches the control information to be output next after output of the control information is completed. The imaging device according to any one of claims 1 to 10, characterized in that the image processing device has the switching means. The imaging device according to claim 11, characterized in that the imaging element and the image processing device each acquire position information of a focus lens and generate control information for the focus lens based on the position information. The imaging device according to claim 11, characterized in that the imaging element does not acquire position information of the focus lens, and the image processing device updates the first control information based on position information of the focus lens and the first control information acquired from the imaging element. the imaging element includes the switching means and an interface capable of outputting the control information to the lens device,
An imaging device as described in any one of claims 1 to 10, characterized in that the image processing device does not acquire position information of a focus lens, and in the imaging element, the second control information is updated based on position information of a focus lens and the second control information acquired from the image processing device. A focus lens for adjusting the focus;
a driving means for driving the focus lens;
A control means for controlling the driving means,
15. The imaging apparatus according to claim 1, wherein the control means controls the drive means to drive the focus lens in accordance with the control information for the focus lens output from the switching means. A method for controlling an imaging device having an imaging element and an image processing device, comprising:
generating first control information for controlling driving of a focus lens in the imaging element;
generating second control information for controlling driving of a focus lens by the image processing device;
switching control information to be output for use in driving a focus lens to either the first control information generated by the image sensor or the second control information generated by the image processing device;
the control information includes information indicating a drive amount of the focus lens,
A method for controlling an imaging device, characterized in that the imaging element has a processor different from the image processing device, and the first control information is generated using a result of subject detection within the imaging device, without using a result of subject detection in the image processing device.

Images