JP7630282B2 - Imaging device and control method thereof, and image processing device and method - Google Patents

Description

The present invention relates to an imaging device and a control method thereof, and an image processing device and method, and in particular to vibration reduction technology in an imaging device.

In recent years, as imaging devices have become more sophisticated, many imaging devices are equipped with a technology that corrects camera shake by moving the image sensor (image stabilization). In this image stabilization technology, many image stabilization mechanisms have been proposed that allow rotational movement around the optical axis in addition to translational movement in a plane perpendicular to the optical axis.

Another type of image stabilization is electronic image stabilization, which corrects image stabilization by electronically aligning the captured image so that it overlaps with a reference image. There is also a technology that aligns and combines multiple images to obtain an image equivalent to a long exposure (hereafter referred to as "image synthesis image stabilization"). Electronic image stabilization does not require image stabilization, and is often used for videos that require real-time performance. On the other hand, image synthesis image stabilization requires stabilization processing and is designed for long exposures, so it is generally used for still images.

Patent Document 1 also discloses a technology that performs image stabilization by coordinating image plane stabilization, which rotates the image sensor to correct rotational shake, with electronic stabilization.

JP 2012-242563 A

In general, when it comes to image stabilization for still images, emphasis is placed on the image quality of each image, while when it comes to image stabilization for videos, emphasis is placed on the chronological stability of the images. As a result, the requirements for image synthesis stabilization for still images and electronic stabilization for videos are different, so if image plane stabilization and electronic stabilization are performed using the same method, the desired image quality may not be obtained for either the still image or the video image.

The present invention was made in consideration of the above problems, and aims to provide image stabilization control suitable for both video and still images.

The second objective is to improve the image quality of the composite image when multiple images are composited.

In order to achieve the above object, an image pickup apparatus according to the present invention comprises: an image pickup means for photoelectrically converting light incident via an image pickup optical system and outputting an image;
The imaging device has a shake detection means which detects shake of an imaging device , a drive means which translates and rotates the imaging means in a plane perpendicular to the optical axis of the imaging optical system, and a generation means which generates a translation amount for the translation drive and a rotation amount for the rotation drive so as to correct the shake detected by the shake detection means, wherein when a plurality of images output from the imaging means are combined, when the plurality of images are captured by the imaging means, the generation means generates a rotation amount for the rotation drive of the imaging means, and generates the translation amount within a range in which the imaging means can be translationally driven after the rotation drive .

The present invention makes it possible to perform image stabilization control suitable for both video and still images. In addition, when multiple images are combined, the image quality of the combined image can be improved.

1 is a central cross-sectional view and a block diagram showing a schematic configuration of an imaging system according to a first embodiment of the present invention. FIG. 2 is an exploded perspective view of the vibration isolation mechanism according to the first embodiment. 4 is a flowchart illustrating processing according to the first embodiment. 5A and 5B are diagrams illustrating the relationship between rotation and translation of the vibration isolation mechanism according to the first embodiment. 13A to 13C are diagrams illustrating tilt shake according to a second embodiment. 10 is a flowchart illustrating a process according to a second embodiment. 13A and 13B are diagrams for explaining the relationship between the amount of rotational shake and tilt shake and the focal length according to the second embodiment. FIG. 13 is a block diagram showing a schematic configuration of an imaging system according to a third embodiment. 13A to 13C are diagrams illustrating the structure of an image sensor according to a third embodiment. 13A to 13C are diagrams for explaining processes performed by an image deviation calculator and a reliability detector according to the third embodiment. 13A and 13B are a schematic external view and a block diagram showing the configuration of an image processing unit of an imaging apparatus according to a fourth embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
1A and 1B are a central sectional view of an imaging system as an imaging apparatus according to an embodiment of the present invention, and a block diagram showing a schematic configuration of the imaging system.

As shown in FIG. 1(a), the imaging system of this embodiment mainly comprises a camera body 1 and a lens unit 2 that is detachable from the camera body 1. The camera body 1 and the lens unit 2 are electrically connected via electrical contacts 11. Note that the imaging device of the present invention is not limited to this configuration, and may be an imaging device in which the camera body and the lens unit are integrally configured.

The lens unit 2 includes a photographing optical system 3 arranged on an optical axis 4 and consisting of multiple lenses, including a focus lens, a zoom lens, an anti-vibration lens, and an aperture, a lens system control circuit 12, and a lens memory 17. The camera body 1 also includes an image sensor 6, a rear display device 9a, an EVF 9b, an anti-vibration mechanism 14, a shake detection unit 15, and a shutter mechanism 16.

Figure 1(b) is a diagram showing the electrical configuration of the imaging system, in which the lens unit 2 further includes a lens drive mechanism 13 that drives the focus lens, zoom lens, anti-vibration lens, aperture, etc. included in the imaging optical system 3. The camera body 1 further includes a camera system control circuit 5, an image processing unit 7, a memory 8, a display unit 9, and an operation detection unit 10. The display unit includes a rear display device 9a and an EVF 9b.

In the imaging system having the above configuration, light from a subject incident through the imaging optical system 3 is imaged on the imaging surface of the imaging element 6. The imaging element 6 photoelectrically converts the incident light and outputs an electrical signal (image signal) according to the amount of light. The imaging element 6 can output image signals of images in various formats, such as so-called still images and videos, and can output videos in multiple formats by changing the aspect ratio, resolution of the recorded image, etc. In addition, the imaging system of this embodiment has a mode (dynamic range expansion, noise reduction, etc.) that acquires and synthesizes a group of images that are consecutive in time for still images, and there are cases where a group of images is acquired consecutively in time, regardless of whether they are still images or videos.

The image processing unit 7 has an internal A/D converter, a white balance adjustment circuit, a gamma correction circuit, an interpolation calculation circuit, etc., and processes the image signal output from the imaging element 6 to generate image data for recording.

The image processing unit 7 is also connected to the camera system control circuit 5. The camera system control circuit 5 can determine the focus evaluation value and exposure amount based on the image signal from the image sensor 6 processed by the image processing unit 7, and can determine the focus position and exposure conditions (F-number, shutter speed, etc.) based on these values. The camera system control circuit 5 then issues a command to the lens system control circuit 12 via the electrical contacts 11 based on the determined focus position and exposure conditions, and the lens system control circuit 12 controls the lens drive mechanism 13 based on the command. The shutter mechanism 16 controls the incidence/blocking of the subject image on the image sensor 6 by moving the shutter curtain. Through the above control, the image sensor 6 is exposed to an appropriate amount of light, and the subject image is formed near the image sensor 6.

The image processing unit 7 in this embodiment also includes a position alignment unit 71 that performs electronic image stabilization by shifting and aligning the positions of multiple images, and an image synthesis unit 72 that synthesizes multiple images. The specific operations of the position alignment unit 71 and the image synthesis unit 72 will be described later. The image processing unit 7 also compresses data such as images, videos, and audio using a predetermined method.

The memory 8 includes an image storage section, and the camera system control circuit 5 outputs to the recording section of the memory 8 .
When the user is looking into the EVF 9b, the display unit 9 turns off the rear display device 9a and presents images and information on the EVF 9b under the control of the camera system control circuit 5, and when the user is not looking into the EVF 9b, the display unit 9 presents images and information on the rear display device 9a. The rear display device 9a is a touch panel and is connected to the operation detection unit 10.

The shake detection unit 15 can detect translational shake in the translation direction on a plane perpendicular to the optical axis 4 and rotational shake of the device in the rotation direction around the optical axis 4, and can use a vibration gyroscope or an acceleration sensor. Shake can also be detected by comparing images between frames based on the image signal output from the imaging element 6. The vibration isolation mechanism 14 is a mechanism that drives the imaging element 6 in translation in a plane perpendicular to the optical axis 4 and rotates it around the optical axis 4, and its specific structure will be described later with reference to FIG. 2.

The camera system control circuit 5 generates and outputs timing signals for imaging. In addition, in response to an external operation detected by the operation detection unit 10, it controls the imaging process, image processing, and recording/playback process. For example, when the operation detection unit 10 detects the pressing of a shutter release button (not shown), the camera system control circuit 5 controls the driving of the image sensor 6, the operation of the image processing unit 7, the operation of the compression process, and the like. The camera system control circuit 5 also includes an alignment on/off unit 51 and an image synthesis on/off unit 52 for turning on/off the operations of the alignment unit 71 and the image synthesis unit 72 included in the image processing unit 7, respectively, and further includes an anti-shake control unit 53. The anti-shake control unit 53 generates a target value for the drive amount of the image sensor 6 by the anti-shake mechanism 14 from the signal of the shake detection unit 15 based on the on/off of the alignment and the on/off of the image synthesis, as described below, and performs anti-shake drive (image surface anti-shake) control.

In addition, in a mode that performs normal image stabilization, the camera system control circuit 5 can also perform known optical image stabilization by controlling the image stabilization lens included in the photographing optical system 3 via the lens drive mechanism 13 based on the signal obtained from the image sensor 6.

Here, we will briefly explain the flow of image stabilization control in an imaging system with the above configuration.

In this embodiment, image stabilization control is performed using a shake detection unit 15 that detects shake, an anti-shake mechanism 14 that performs image stabilization, and an anti-shake control system provided in the camera system control circuit 5. Then, during the so-called composition determination operation, in which the operation detection unit 10 detects an operation (SW1) of pressing the shutter release button (not shown) halfway down to enter a preliminary shooting operation, image stabilization is performed using the anti-shake mechanism 14 to make it easier to determine the composition. That is, image stabilization is performed by controlling the anti-shake mechanism 14 based on a signal from the shake detection unit 15. After that, when the operation detection unit 10 detects an operation (SW2) of pressing the shutter release button completely down to enter a shooting operation, image stabilization is performed using the anti-shake mechanism 14 to suppress blurring of the subject image obtained by exposure. When a certain time has passed after exposure, the image stabilization operation is stopped.

Next, the vibration isolation mechanism 14 of this embodiment will be described with reference to FIG. 2. FIG. 2 is an exploded perspective view of the vibration isolation mechanism 14, which is a mechanism that performs blur correction. Note that there is a separate electrical mechanism for control, but this is not included in the diagram shown in FIG. 2. In FIG. 2, the vertical lines are parallel to the optical axis. In FIG. 2, non-moving members (fixed parts) are numbered in the 100s. Moving members (movable parts) are numbered in the 200s. Furthermore, the balls that are sandwiched between the fixed and movable parts are numbered in the 300s.

In FIG. 2, 101 is an upper yoke, 102a, 102b, and 102c are screws, 103a, 103b, 103c, 103d, 103e, and 103f are upper magnets, 104a and 104b are auxiliary spacers, and 105a, 105b, and 105c are main spacers. In addition, 106a, 106b, and 106c are fixed part rolling plates, 107a, 107b, 107c, 107d, 107e, and 107f are lower magnets, 108 is a lower yoke, 109a, 109b, and 109c are screws, and 110 is a base plate.

201 is the FPC, 202a, 202b, and 202c are the mounting positions of the position detection element, 203 is the movable PCB, 204a, 204b, and 204c are the moving part rolling plates, 205a, 205b, and 205c are coils, 206 is the movable frame, and 301a, 301b, and 301c are balls.

The upper yoke 101, the upper magnets 103a, 103b, 103c, 103d, 103e, 103f, the lower magnets 107a, 107b, 107c, 107d, 107e, 107f, and the lower yoke 108 form a magnetic circuit, which is a so-called closed magnetic circuit. The upper magnets 103a, 103b, 103c, 103d, 103e, 103f are glued and fixed in a state where they are attracted to the upper yoke 101. Similarly, the lower magnets 107a, 107b, 107c, 107d, 107e, 107f are glued and fixed in a state where they are attracted to the lower yoke 108. The upper magnets 103a, 103b, 103c, 103d, 103e, and 103f and the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f are each magnetized in the optical axis direction (the vertical direction in FIG. 2). Furthermore, adjacent magnets (those in the positional relationship of magnets 103a and 103b) are magnetized in different directions. Furthermore, opposing magnets (those in the positional relationship of magnets 103a and 107a) are magnetized in the same direction. In this way, a strong magnetic flux density is generated in the optical axis direction between the upper yoke 101 and the lower yoke 108.

Since a strong attractive force occurs between the upper yoke 101 and the lower yoke 108, the main spacers 105a, 105b, 105c and the auxiliary spacers 104a, 104b are configured to maintain an appropriate distance. The appropriate distance here is a distance that allows the coils 205a, 205b, 205c and the FPC 201 to be placed between the upper magnets 103a, 103b, 103c, 103d, 103e, 103f and the lower magnets 107a, 107b, 107c, 107d, 107e, 107f while ensuring an appropriate gap. The main spacers 105a, 105b, 105c have screw holes, and the upper yoke 101 is fixed to the main spacers 105a, 105b, 105c by the screws 102a, 102b, 102c.

Rubber is installed on the body of the main spacers 105a, 105b, and 105c, forming the mechanical ends (so-called stoppers) of the moving parts.

The base plate 110 has holes to avoid the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f, and is configured so that the faces of the magnets protrude from the holes. In other words, the base plate 110 and the lower yoke 108 are fixed with screws 109a, 109b, and 109c, and the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f, which have a larger dimension in the thickness direction than the base plate 110, are fixed so as to protrude from the base plate 110.

The movable PCB 203 is made of magnesium die-cast or aluminum die-cast, and is lightweight and highly rigid. Each element of the movable part is fixed to the movable PCB 203 to form the movable part. Position detection elements are attached to the FPC 201 on the side that cannot be seen in FIG. 2 at the positions indicated by position detection element attachment positions 202a, 202b, and 202c. For example, a Hall element or the like can be used to detect the position using the magnetic circuit described above. Since the Hall element is small, it is placed so that it is nested inside the windings of the coils 205a, 205b, and 205c.

The movable PCB 203 is connected to an imaging element 6 (not shown), coils 205a, 205b, 205c, and a Hall element. Electrical communication with the outside is performed via a connector on the movable PCB 203.

Fixed part rolling plates 106a, 106b, and 106c are glued to the base plate 110, and movable part rolling plates 204a, 204b, and 204c are glued to the movable PCB 203, forming the rolling surfaces of the balls 301a, 301b, and 301c. By providing the rolling plates separately, it becomes easier to design the surface roughness, hardness, and other properties to the desired state.

By passing a current through coils 205a, 205b, and 205c in the above-mentioned configuration, a force according to Fleming's left-hand rule is generated, and the movable frame 206 can be moved. In addition, feedback control can be performed by using the signal from the Hall element, which is the position detection element mentioned above. By appropriately controlling the value of the Hall element signal, it is possible to perform translational motion in a plane perpendicular to the optical axis and rotation around the optical axis.

By driving the Hall element signals at the position detection element mounting positions 202b and 202c in opposite phases while keeping the signal of the Hall element at the position detection element mounting position 202a constant, it is possible to generate rotational motion approximately around the optical axis 4.

The magnetic flux density in the optical axis direction is detected at the position detection element attachment positions 202a, 202b, and 202c. The characteristics of the magnetic circuit consisting of the upper magnets 103a, 103b, 103c, 103d, 103e, and 103f and the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f are generally nonlinear. Therefore, the magnetic flux density detected at the position detection element attachment positions 202a, 202b, and 202c does not necessarily have a constant resolution throughout the entire driving range, so the detection resolution changes. This is because there are positions where the magnetic flux density changes steeply and positions where it changes gently, and the steeper the position, the higher the detection resolution is, that is, the larger the change in magnetic flux density relative to the amount of movement. In the magnetic circuit shown in FIG. 2, the change in magnetic flux density is the largest at the boundary position of the magnets (for example, the boundary position between the upper magnets 103a and 103b), and the detection resolution is high.

Since many proposals have been made regarding methods for controlling the vibration isolation mechanism 14, a detailed explanation will be omitted here.

Next, the processing performed by the camera body 1 having the above configuration will be explained using the flowchart shown in FIG.

In S301, the image synthesis on/off unit 52 in the camera system control circuit 5 controls whether or not to perform image synthesis. Note that the user may be able to directly instruct whether or not to perform image synthesis, or it may be automatically switched depending on the camera settings such as the shooting mode and the shooting conditions such as the shutter speed. For example, image synthesis is performed when the shutter speed is slower than the threshold in the still image shooting mode, and image synthesis is not performed when the shutter speed is faster than the threshold in the still image shooting mode or in the video shooting mode. Note that image synthesis is performed after the alignment unit 71 performs alignment, so when image synthesis is performed, the alignment on/off unit 51 is also turned on.
If image synthesis is to be performed, the process proceeds to S302. If image synthesis is not to be performed, the process proceeds to S303.

In S302, the image stabilization control unit 53 of the camera system control circuit 5 controls image stabilization so that, out of translational shake and rotational shake (roll shake), rotational shake is given priority in correction. Hereinafter, this control will be referred to as "rotational shake correction priority control." Rotational shake is given priority in correction in order to suppress deterioration in image quality that occurs during image alignment in S304 and image synthesis in S305, which will be described later. This control is suitable for performing image synthesis stabilization accompanied by image alignment in still image shooting, where image quality is important.

Now, the relationship between rotation and translation of the vibration isolation mechanism 14 in this embodiment will be described with reference to FIG. 4. FIG. 4 is a view of the vibration isolation mechanism 14 shown in FIG. 3 as seen from the direction of the optical axis 4 (the Z-axis direction of the coordinate system in the lower right). Note that the coordinate system in the lower right of FIG. 4 is common to FIGS. 4(a), (b), and (c). FIG. 4(a) shows a state in which the movable frame 206 is not moving, FIG. 4(b) shows a state in which the movable frame 206 has moved in the positive X direction without rotation, and FIG. 4(c) shows a state in which the movable frame 206 has moved in the positive X direction with rotation.

The dashed and dotted line 400, which is drawn in common in Figures 4(a), (b), and (c), is a reference line indicating the reference of the fixed part in the X direction, and the dashed line 401 is a reference line indicating the reference of the fixed part in the Y direction. In Figure 4(b), the dashed and dotted line 402 indicates the reference line after the movable frame 206 has moved in the positive X direction. In Figure 4(c), the dashed line 403 indicates the reference line after the movable frame 206 has rotated, and the dashed and dotted line 404 indicates the reference line after the movable frame 206 has moved in the positive X direction. In Figure 4(a), the reference line of the movable part 206 overlaps with the reference line of the fixed part. In other words, it is in a non-moving state.

Figure 4(b) shows the state in which the movable frame 206 has been moved in the positive X direction until it contacts the main spacer 105a without rotating the movable frame 206. In other words, the movable frame 206 can move in the positive X direction an amount equivalent to the distance between the reference line 400 and the reference line 402.

Figure 4(c) shows the state in which the movable frame 206 has been rotated and moved in the positive X direction until it comes into contact with the main spacer 105a. The amount of rotation is indicated by the angle between reference line 401 and reference line 403. The amount of movement of the movable frame 206 in the X direction at this time corresponds to the distance between reference line 400 and reference line 404.

As is clear from a comparison of Figures 4(b) and 4(c), the amount of translation possible in the X direction changes when rotation occurs. To suppress this change, the mechanical end of the movable part 206 could be placed on the optical axis 4, but this is not easy due to the presence of the image sensor 6 and the processing board behind it. For this reason, the mechanical end of the movable part (main spacers 105a, 105b, and 105c in the examples of Figures 2 and 4) is generally placed at a position other than the optical axis 4.

As is clear from FIG. 4(c), the gap between the main spacer 105c and the movable frame 206 in the Y direction changes compared to when there is no rotation. In other words, it can be seen that the amount of movement possible in the positive Y direction also decreases as a result of rotation.

Although Figures 4(b) and (c) do not show the specific relationship between the amount of rotation and the amount of translation, it is clear from Figure 4(c) that, in general, as the amount of rotation increases, the amount of translation that can be moved decreases. Furthermore, to find a specific relationship, the position of the mechanical end of the movable part can be determined and a numerical calculation can be performed. In order to have a margin for the amount of rotation and the amount of translation, the main spacers 105a, 105b, and 105c can be designed away from the movable frame 206, but this will lead to an increase in the size of the device. Since the camera body 1 is carried around by the user, it is very important not to increase the size of the device.

As described above, since the amount of rotation and the amount of translation are in a contradictory relationship, in S302, the upper limit of the amount of rotational shake compensation (amount of rotational drive) is set higher than the upper limit of the amount of translational shake compensation (amount of translational drive), thereby giving priority to correcting rotational shake.

In S304, the alignment unit 71 of the image processing unit 7 detects the misalignment between the multiple images input from the image sensor 6, and performs geometric deformation on the images to correct the misalignment.

Positional deviation between images can be detected by calculating the similarity between the images using known template matching methods. In geometric transformation, the images are subjected to affine or projective transformation to correct positional deviation in the translation and rotation directions of the images. This alignment achieves electronic image stabilization.

In addition, in a geometric transformation, it is necessary to associate each pixel of an image before and after the transformation. When an image is translated or rotated in sub-pixel units, each pixel does not correspond one-to-one, so the correspondence is generally achieved by using pixels interpolated from multiple surrounding pixels.
However, when this pixel interpolation is performed, the image resolution is lost, causing degradation of image quality. Therefore, in order to prevent degradation of image quality, it is desirable to avoid translational movement and rotational movement in sub-pixel units as much as possible. Regarding translational movement, by limiting movement in integer pixel units, it is possible to avoid degradation of image quality due to pixel interpolation, although the alignment accuracy decreases. Furthermore, if the rotational shake correction priority control in S302 is used, rotational shake is sufficiently corrected by the vibration isolation mechanism 14, making rotational movement unnecessary and preventing degradation of image quality.

In S305, the image synthesis unit 72 of the image processing unit 7 synthesizes the multiple images that were aligned in S304 by adding them together. This achieves image synthesis and image stabilization. Note that if each of the multiple images is shot with the correct exposure, it is necessary to take the average after adding the images together.

On the other hand, if image synthesis is not performed, in S303, the image stabilization control unit 53 controls image plane stabilization so as to correct translational shake and rotational shake in a well-balanced manner. Hereinafter, this control will be referred to as "standard shake correction control." This control is suitable for stabilization in video shooting, where time-series stability is important. It is also suitable for still image shooting, where image alignment is not performed, i.e., where image synthesis stabilization is not performed.

In standard shake correction control, the upper limit of the amount of rotational shake correction is set lower than in the rotational shake correction priority control in S302, and is set to be approximately the same as the upper limit of the amount of translational shake correction, thereby achieving a good balance between correction of translational shake and rotational shake.

Next, in S306, the alignment on/off unit 51 of the camera system control circuit 5 controls whether or not to perform image alignment. The user may be able to set directly whether or not to perform image alignment, or the setting may be switched automatically depending on the camera settings and shooting conditions. For example, image alignment may be performed in video shooting mode, and image alignment may not be performed if the shutter speed is faster than a threshold in still image shooting mode.

If image alignment, i.e. electronic stabilization, is to be performed, proceed to S307; if electronic stabilization is not to be performed, end the process.

In S307, the same image alignment process as in S304 is performed, and the process ends.

As described above, according to the first embodiment, when image synthesis anti-shake is performed, image plane anti-shake is controlled with rotational shake correction priority, thereby enabling anti-shake suitable for still images that emphasize image quality. Also, when image synthesis anti-shake is not performed, image plane anti-shake is controlled with standard shake correction control, thereby enabling anti-shake suitable for moving images that emphasize stability.

In the above embodiment, an example was described in which the rotational shake correction priority control is performed by setting the upper limit of the rotational shake correction amount (rotational drive amount) higher than the upper limit of the translational shake correction amount (translation drive amount), thereby giving priority to correcting the rotational shake. However, the method of giving priority to correcting the rotational shake over the translational shake is not limited to this. For example, the rotational shake correction amount may be calculated first, and the translational shake may be corrected within the remaining movable range. In addition, when the total of the rotational shake correction amount and the translational shake correction amount exceeds the drivable range, the gain used when obtaining the rotational shake correction amount based on the detected rotational shake may be made larger than the gain used when obtaining the translational shake correction amount based on the detected translational shake, thereby giving priority to correcting the rotational shake. The correction amount may be set so that the detected shake is corrected 100% when the gain is 1.

Second Embodiment
Next, a second embodiment of the present invention will be described.
In the first embodiment, the vibration isolation mechanism 14 is a mechanism that translates the image sensor 6 in a plane perpendicular to the optical axis 4 and rotates it about the optical axis 4. In the second embodiment, in addition to the configuration shown in Fig. 4, the vibration isolation mechanism 14 further has a mechanism that rotates it about two axes perpendicular to the optical axis 4. This mechanism makes it possible to correct not only translational shake and rotational shake, but also tilt components caused by shake in the yaw and pitch directions (hereinafter referred to as "tilt shake") in the image plane vibration isolation. Note that the overall configuration of the imaging system is the same as that shown in Fig. 1, so a description thereof will be omitted.

Figure 5 is a diagram explaining tilt shake. Figure 5(a) shows the state when shaking in the yaw direction occurs. 501 shows the original image, and 502 shows the image after shaking in the yaw direction. When shaking in the yaw direction occurs in this way in the camera, tilt shake α occurs in addition to translational shake Tx in the image, causing it to be distorted into a trapezoid shape.

Similarly, Figure 5(b) shows what happens when shaking occurs in the pitch direction. 501 shows the original image, and 503 shows the image after shaking in the yaw direction. When shaking occurs in the pitch direction in this way, the image is distorted into a trapezoidal shape due to tilt shake β in addition to translation shake Ty.

Next, the image plane stabilization operation according to the second embodiment will be described with reference to the flowchart shown in FIG. 6. Note that in FIG. 6, the same processes as those shown in FIG. 3 are given the same step numbers, and descriptions thereof will be omitted as appropriate.

In this second embodiment, when performing image synthesis, in S601, the image stabilization control unit 53 of the camera system control circuit 5 controls image stabilization so as to give priority to correcting a specific type of shake among translational shake, rotational shake, and tilt shake. Hereinafter, this control will be referred to as "specific shake correction priority control."

When trying to correct tilt shake during image alignment, pixel interpolation occurs, just as with rotational shake, resulting in degradation of image quality. For this reason, when using image synthesis image stabilization, it is preferable to correct tilt shake in addition to rotational shake using image plane stabilization.

The priority of correction for rotational shake and tilt shake differs depending on the focal length. Figure 7 shows the relationship between the amount of rotational shake and tilt shake and the focal length. In areas where the focal length is shorter than ft, tilt shake becomes dominant, and in areas where the focal length is longer than ft, rotational shake becomes dominant. Therefore, when the focal length is shorter than a predetermined focal length, correction is prioritized in the order of tilt shake > rotational shake > translational shake, and when the focal length is equal to or greater than the predetermined focal length, correction is prioritized in the order of rotational shake > tilt shake > translational shake.

As described above, according to the second embodiment, image plane stabilization corrects tilt shake in addition to rotational shake, making it possible to perform image synthesis stabilization with less degradation in image quality.

Third Embodiment
Next, a third embodiment of the present invention will be described.
When correcting camera shake by synthesizing multiple images as in the first and second embodiments, if the images are not aligned properly before synthesis and are misaligned, a double image will be generated. Therefore, in this embodiment, a case where the reliability of the alignment is determined will be described.

Figure 8 is a diagram showing a schematic configuration of an imaging system in the third embodiment. Compared to the configuration of the imaging system shown in Figure 1 (b), it differs in that the image processing unit 7' further includes an image shift calculator 74 and a reliability detector 75. The rest of the configuration is the same as that described with reference to Figures 1 and 2, so description thereof will be omitted.

Figure 9 is a diagram showing the configuration of the image sensor 6 in the third embodiment. Figure 9(a) is an enlarged view of some pixels 20 (4x4 pixels) of the image sensor 6 when viewed from the optical axis 4 direction, and Figure 9(b) is a diagram showing a cross-sectional view (lower part) of one pixel 20 and its correspondence with the exit pupil of the photographing lens. Note that although the projection directions are different in the lower and upper parts of Figure 9(b), this method of illustration is used to make it easier to understand the correspondence state on the pupil plane.

In the example shown in FIG. 9(a), the image sensor 6 is covered with color filters in a so-called Bayer array, with R, G, and B representing the colors of the color filters red, green, and blue, respectively. The circles in each pixel 20 represent microlenses 25, and the vertical lines in each pixel 20 indicate that the light receiving section of each pixel 20 is divided into two light receiving sections 22a and 22b (photoelectric conversion sections) in the X direction. The arrows 21 indicate the distance between pixels covered with filters of the same color.

In FIG. 9(b), 23a and 23b denote areas on the pupil plane of the photographing optical system 3 that correspond to the light receiving units 22a and 22b, respectively, and 24 denotes the exit pupil of the photographing optical system 3.

As shown in FIG. 9(b), the light receiving sections 22a and 22b of the pixel 20 are conjugated to regions 23a and 23b of the pupil plane of the imaging optical system 3 by the microlens 25, respectively, and are configured to receive light beams that have passed through different pupil regions of the exit pupil 24.

From the imaging element 6 having the above configuration, signals having parallax (or signals from different viewpoints) obtained by each of the light receiving units 22a and 22b are acquired, and a phase difference calculation is performed to perform a known phase difference AF. The phase difference AF calculation is performed by the camera system control circuit 5. Also, a normal pixel signal can be obtained by adding and outputting the signals obtained by the light receiving units 22a and 22b for each pixel 20. Note that in FIG. 9(b), the areas 23a and 23b are shown with a gap to clearly indicate that they are different areas, but in reality they may be in contact or may have overlapping areas. Hereinafter, the image obtained by collecting the pixel signals obtained from the light receiving units 22a of each pixel 20 will be called the "A image", and the image obtained by collecting the pixel signals obtained from the light receiving units 22b of each pixel 20 will be called the "B image".

Next, the processes performed by the image deviation calculator 74 and the reliability detector 75 in the third embodiment will be described with reference to FIG.
The horizontal axis in Fig. 10(a) represents time. The time when acquisition begins is designated as _t1 , and thereafter, images A and B are acquired successively at _t2 , t3 _, and so on, in a predetermined cycle. The image A acquired at time t is designated _{as A1} , and the image B is designated as _B1 . Similarly, the image A acquired at time _tn is designated _as An, and the image B is designated as _Bn .

10(b) and (c) show an example in which a circulation path is formed between the images A and B acquired after time _t2 and the images A and B acquired previously. Fig. 10(d), as a comparison with Fig. 10(b) and (c), shows an example in which a circulation path is not formed between the images A and B acquired after time _t2 and the images A and B acquired previously.

10B to 10D, consider images A and B acquired at times t _n-1 and t _n . In FIG. 10B to 10D, images A and B acquired at time t _n-1 are used as templates for calculating a motion vector. However, if the image at time t n _-1 is rejected as a template at the previous time, an image before time t _n-1 may become the template. The rejection of templates will be described later.

In FIG. 10B, at the timing of time t _n , a movement vector 30 from A _n to B _n , a movement vector 31 from B _n to B _n-1 , and a movement vector 33 from A _n-1 to A _n are calculated. As can be seen from the figure, by repeating the calculation of the above movement vectors, the movement vector 32 from B _n-1 to A _n-1 can be used by inverting the sign because the movement vector in the opposite direction is calculated at the time of time t _n-1 . What is important here is that the four movement vectors 30, 31, 32, and 33 form a path that returns to A _n without departing from A _n and following the same path. In this embodiment, this is called "forming a circular path." In the example of FIG. 10B, it is clear that a circular path is formed between the image group (A _n , B _n ) acquired after time t ₂ and the image group (A _n-1 , B _n-1 ) acquired before that time.

10C, at time t _n , a motion vector 30 from A _n to B _n , a motion vector 34 from B _n to A _n-1 , and a motion vector 56 from B _n-1 to A _n are calculated. Note that motion vector 35 is calculated at time t _n-1 . As can be seen from the figure, these four motion vectors 30, 34, 35, and 36 form a circulating path. The image deviation detector 54 of this embodiment operates to form a circulating path in this manner.

As shown in the examples of Figures 10(b) and (c), there are multiple possible ways to form a circulation path, and any convenient method can be selected. However, comparisons between images A and B, or between images acquired at the same time, are convenient because they are stable with respect to vignetting conditions and the state of the subject. In other words, the example shown in Figure 10(b) can be said to be a more preferable example.

Fig. 10D shows the operation of image shift detection that is normally performed. Unlike this embodiment, in the example of Fig. 10D, an image (A _n +B _n ) is generated by adding A _n and B _n at the timing of time t _n . Then, a movement vector is calculated between the template image A _n-1 +B _n-1 . As is clear from Fig. 10D , no path that allows circulation is formed.

Next, we will explain how to use the circular path. In this embodiment, the reliability detector 75 circulates and accumulates the movement vector to the original image again. If there is no detection error, the accumulation result will be zero since it returns to the original position. On the other hand, if detection fails for some reason, such as the movement of the subject, a mistake in a repeating pattern, or a change in the ambient light, the result will not be zero.

Therefore, if the result of the reliability detector 75 is greater than the threshold, it is determined that a detection error has occurred. This makes it possible to determine at time _tn whether the movement vector found at time _tn is good or bad. When aligning and synthesizing images, it is possible to determine whether to add them based on the good or bad judgment at this point, thereby saving memory.

Furthermore, based on a comparison with a threshold value, it is possible to determine whether or not to update the template. If it is equal to or less than the threshold, the image at time _tn is used as a template for the image to be acquired at the next time, and if it is greater than the threshold, the template is not updated. This makes it possible to perform stable synthesis processing by excluding a single frame containing an image with different characteristics (e.g., positioning failure due to flickering).

In the above example, the light receiving portion of each pixel 20 is divided into two light receiving portions 22a and 22b in the X direction, but the present invention is not limited to this. For example, it may be divided in the Y direction, or the division directions may be mixed. Furthermore, the number of divisions is not limited to two, but may be three or more. In any case, the reliability of the alignment can be determined by using the method of using the movement vector between the images, as described above.

In this embodiment, the image shift calculator 74 and the reliability detector 75 are provided in the imaging device, but the present invention is not limited to this. As another configuration, for example, the image shift calculator 74 and the reliability detector 75 may be used to store the image group A and the image group B, and process them later, using the optical system as shown in this embodiment. In this case, the image shift calculator 74 and the reliability detector 75 are realized in the form of a computer program or the like, and the stored image group A and the image group B are processed. The processing content is as shown in this embodiment, and the image shift amount is calculated for the image group after time _t2 so as to form a cyclic path between the image group before that time and the image group, and the detection amount along the cyclic path is integrated. By using this result, it is possible to determine whether the movement vector calculated at time _tn is good or bad at the timing of time _tn . In this case, as in the example of the imaging device, memory can be saved and memory access can be reduced, allowing high-speed calculation.

As described above, according to the third embodiment, it is possible to determine the reliability of alignment with a simple configuration.

Fourth Embodiment
Next, a fourth embodiment of the present invention will be described.
Figure 11(a) is a schematic external view showing a smartphone 41 as an imaging device in the fourth embodiment, and Figure 11(b) is a block diagram mainly showing the configuration of an image processing unit 7". As the smartphone 41 in the fourth embodiment, a so-called multi-lens camera having two shooting optical systems 3a, 3b and two image sensor 6a, 6b will be described. The shooting optical systems 3a, 3b have different focal lengths and can capture images with different angles of view, and here, it is assumed that the shooting optical system 3b has a shorter focal length. In addition, since the shooting optical systems 3a and 3b are spaced apart, images can be captured from multiple viewpoints simultaneously. 4a and 4b indicate the optical axes of the shooting optical systems 3a and 3b, respectively.

Other than the above, the circuit configuration is similar to that shown in FIG. 1(b), but the processing in the image processing unit 7" is different, which will be explained below. In this embodiment, the image processing unit 7" includes a magnification adjuster 73 in addition to the configuration of the image processing unit 7' in FIG. 8.

In this embodiment, the image obtained from the imaging element 6a is referred to as image A. On the other hand, the image obtained from the imaging element 6b is sent to the magnification adjuster 73 and converted into a state that can be compared with the imaging element 6a. Specifically, a part of the image obtained at a wider angle is cropped and processed to match the pixel pitch. The output image of this magnification adjuster 73 is referred to as image B.

Then, the acquired images A and B are processed as described in the third embodiment with reference to FIG. 10 using the images A and B. That is, the image shift calculator 74 calculates the amount of image shift for the second and subsequent image groups so as to form a circular path between the image groups that precede them. Then, the reliability detector 75 accumulates the detection amount along the circular path. As a result, the same effect as in the third embodiment can be obtained.

In this embodiment, the smartphone 41 has been described as having two imaging optical systems 3a and 3b and two image sensors 6a and 6b, but the number of imaging systems may be three or more. In that case, it is sufficient to provide the required number of magnification adjusters 73.

Furthermore, this embodiment is not limited to smartphones, but can be applied to any electronic device capable of processing images.

The present invention can also be applied to a system consisting of multiple devices, or to a device consisting of a single device.

<Other embodiments>
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions.

The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

1: Camera body, 2: Lens unit, 3: Shooting optical system, 4: Optical axis, 5: Camera system control circuit, 6: Image sensor, 7: Image processing unit, 10: Operation detection unit, 14: Anti-vibration mechanism, 15: Shake detection unit, 51: On/off unit, 52: Image synthesis on/off unit, 53: Anti-vibration control unit, 71: Positioning unit, 72: Image synthesis unit, 73: Magnification adjuster, 74: Image deviation calculator, 75: Reliability detector

Claims (19)

an imaging means for photoelectrically converting light incident via an imaging optical system to output an image;
A shake detection means for detecting a shake of the imaging device;
a driving means for translating and rotating the imaging means in a plane perpendicular to an optical axis of the photographing optical system;
a generating unit that generates a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected by the shake detection unit ,
an imaging device characterized in that, when a plurality of images output from the imaging means are combined, when the plurality of images are captured by the imaging means, the generation means generates a rotation amount for driving the imaging means in a rotational manner, and generates the translation amount within a range in which the imaging means can be driven in a translational manner after the rotational drive. 2. The imaging apparatus according to claim 1 , wherein the driving means sets the upper limit of the rotational drive higher when the plurality of images are combined than when the combination is not performed. an imaging means for photoelectrically converting light incident via an imaging optical system to output an image;
A shake detection means for detecting a shake of the imaging device;
a driving means for translating and rotating the imaging means in a plane perpendicular to an optical axis of the photographing optical system;
a generating unit that generates a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected by the shake detection unit ,
an imaging device characterized in that, when multiple images output from the imaging means are combined, the generation means sets an upper limit value of the rotational drive higher and an upper limit value of the translational drive lower when capturing the multiple images by the imaging means than when the combination is not performed. A setting means for setting whether or not the plurality of images are to be synthesized,
4. The imaging device according to claim 1, wherein the setting means sets the multiple images to be combined when a user instructs the image to be combined or when a shutter speed in a still image shooting mode is slower than a threshold value. A setting means for setting whether or not the plurality of images are to be synthesized,
The imaging device according to any one of claims 1 to 4, characterized in that the setting means sets the image not to be combined when the user does not instruct image combination, when the shutter speed is faster than a threshold in still image shooting mode, or when the imaging device is in video shooting mode. The driving means further drives the lens in a rotational direction around an axis perpendicular to the optical axis,
6. The imaging device according to claim 1, wherein, when the multiple images are combined and the multiple images are captured by the imaging means, the generation means generates a rotation amount for performing the rotational drive and the tilt drive of the imaging means, generates the translational amount within a range in which the imaging means can be driven in the translational direction after the rotational drive and the tilt drive, generates a rotation amount for the tilt drive when a focal length of the photographing optical system is shorter than a predetermined focal length, generates the rotation amount for the rotational drive within a range in which the imaging means can be driven in the rotational direction after the tilt drive , generates the rotation amount for the rotational drive when the focal length is equal to or greater than the predetermined focal length, and generates the rotation amount for the tilt drive within a range in which the imaging means can be driven in the tilt direction after the rotational drive . a position alignment unit that aligns the plurality of images output from the imaging unit when the plurality of images are to be synthesized;
a synthesis means for synthesizing the plurality of images aligned by the alignment means;
7. The imaging device according to claim 1, further comprising: the imaging means is capable of outputting a first image and a second image having different viewpoints based on light incident via an imaging optical system,
a calculation means for calculating a movement vector indicating an amount of deviation between a plurality of first images obtained at a first timing and a second timing prior to the first timing, and a plurality of second images obtained at the first timing and the second timing, which have been aligned by the alignment means;
a reliability detection unit that detects the reliability of the alignment by the alignment unit by integrating the movement vector obtained by the calculation unit,
8. The imaging device according to claim 7, characterized in that, when the reliability detected by the reliability detection means is lower than a predetermined threshold, the first image and the second image obtained at the first timing are not used for synthesis by the synthesis means. 9. The imaging apparatus according to claim 8, wherein the calculation means forms a circulating path of the movement vector among the first images and the second images aligned by the alignment means. the calculation means calculates a first motion vector from the first image obtained at the first timing to the second image obtained at the second timing, a second motion vector from the second image obtained at the first timing to the second image obtained at the second timing, a third motion vector from the second image obtained at the second timing to the first image, and a fourth motion vector from the first image obtained at the second timing to the first image obtained at the first timing, after the alignment by the alignment means;
10. The imaging apparatus according to claim 9 , wherein the reliability detection means detects the reliability by integrating the first to fourth movement vectors. the calculation means calculates a first motion vector from the first image obtained at the first timing to the second image after alignment by the alignment means, a second motion vector from the second image obtained at the first timing to the first image obtained at the second timing, a third motion vector from the first image obtained at the second timing to the second image, and a fourth motion vector from the second image obtained at the second timing to the first image obtained at the first timing;
9. The imaging apparatus according to claim 8 , wherein the reliability detection means detects the reliability by integrating the first to fourth movement vectors. the imaging means includes an imaging element having a plurality of pixels, each including a first photoelectric conversion unit and a second photoelectric conversion unit that photoelectrically convert light that has passed through different pupil regions of the photographing optical system;
12. The imaging device according to claim 8, wherein the first image corresponds to an image signal photoelectrically converted by the first photoelectric conversion unit, and the second image corresponds to an image signal photoelectrically converted by the second photoelectric conversion unit. the imaging means includes a first imaging element and a second imaging element that photoelectrically convert light incident thereon via a plurality of photographing optical systems,
12. The imaging device according to claim 8, wherein the first image corresponds to an image signal photoelectrically converted by the first imaging element, and the second image corresponds to an image signal photoelectrically converted by the second imaging element. an acquisition means for acquiring a first image and a second image having different viewpoints obtained based on light incident via an imaging optical system from an imaging means;
a positioning unit that performs positioning between the first images obtained by the acquisition unit at a first timing and a second timing prior to the first timing, and performs positioning between the second images obtained at the first timing and the second timing;
a synthesis means for synthesizing the first images and the second images aligned by the alignment means;
a calculation means for calculating a movement vector indicating an amount of deviation between the plurality of first images and the plurality of second images aligned by the alignment means;
a reliability detection means for detecting the reliability of the alignment by the alignment means by integrating the movement vector obtained by the calculation means,
An image processing device characterized in that, when the reliability detected by the reliability detection means is lower than a predetermined threshold, the first image and the second image obtained at the first timing are not used for synthesis by the synthesis means. an imaging step in which the imaging means photoelectrically converts light incident via the imaging optical system and outputs an image;
a blur detection step of detecting blur of an imaging device;
a driving step in which a driving unit drives the imaging unit in a translational direction on a plane perpendicular to an optical axis of the photographing optical system and in a rotational direction around the optical axis;
a generating step of generating a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected in the shake detection step ,
A control method for an imaging device, characterized in that when multiple images output in the imaging process are synthesized and the multiple images are captured by the imaging means, the generation process generates a rotation amount for driving the imaging means in rotation, and generates the translation amount within a range in which the imaging means can be driven in translation after the rotational drive . an imaging step in which the imaging means photoelectrically converts light incident via the imaging optical system and outputs an image;
a blur detection step of detecting blur of an imaging device;
a driving step of driving the imaging means in a translational direction on a plane perpendicular to an optical axis of the photographing optical system and in a rotational direction around the optical axis;
a generating step of generating a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected in the shake detection step ,
A control method for an imaging device, characterized in that, in the generation process, when a plurality of images output from the imaging means are combined, when the plurality of images are photographed by the imaging means, an upper limit value of the rotation amount of the rotational drive is set higher and an upper limit value of the translation amount of the translation drive is set lower than when the combination is not performed. an acquisition step in which an acquisition means acquires, from an imaging means, a first image and a second image having different viewpoints obtained based on light incident via an imaging optical system;
a positioning step in which, when the first images and the second images obtained from the imaging means are combined, a positioning means performs positioning between the first images obtained at a first timing and a second timing that is earlier than the first timing in the acquisition step, and performs positioning between the second images obtained at the first timing and the second timing;
a combining step of combining the first images and the second images, the first images and the second images being aligned in the alignment step, when the combining means combines the first images and the second images;
a calculation step in which a calculation means calculates a movement vector indicating an amount of deviation between the plurality of first images and the plurality of second images aligned in the alignment step;
a reliability detection step in which a reliability detection means detects the reliability of the alignment by the alignment means by integrating the movement vector obtained in the calculation step,
An image processing method characterized in that, when the reliability detected in the reliability detection process is lower than a predetermined threshold, the first image and the second image obtained at the first timing are not used for synthesis in the synthesis process. A program for causing a computer to function as each of the means of the image processing device according to claim 14 . A program for causing a computer to execute each step of the method for controlling an imaging apparatus according to claim 15 or 16 .

Images