JP4579098B2 - Imaging device - Google Patents

Description

  The present invention relates to an image pickup apparatus using a solid-state image pickup device such as a CCD image sensor, and relates to a still image pickup apparatus having a blur correction function for a still image.

  It is a well-known phenomenon that when taking an image of a subject, even if the subject is stationary, the captured image is blurred by moving the imaging device when the photographer presses the shutter button. Is generally expressed as “camera shake”. The reproduction frequency of “camera shake” increases as the illuminance of the environment in which the subject is imaged decreases and the shutter speed decreases (the charge accumulation time increases).

  A technique for improving the quality of a captured image by correcting the captured image due to “shake” mechanically or by image processing to reduce image degradation due to “shake” has been widely studied. For example, by detecting a displacement (angle) of the imaging apparatus using a sensor that detects angular velocity or angular acceleration, and moving the optical lens or the solid-state imaging device itself in a direction to cancel the displacement, “camera shake” is corrected. There is technology.

  For example, an image blur detection unit using an angular velocity sensor is provided, and a unit for driving and controlling the correction optical system in a direction to reduce or eliminate the image blur according to the camera shake detection result and the position detection result of the correction optical system is provided. An imaging device including a correction device is known (see, for example, Patent Document 1).

  In another conventional imaging apparatus, camera shake correction processing by image processing is performed. For example, the charge accumulation time is controlled to give a plurality of consecutive exposures, and a plurality of obtained images are added to generate a single image. There is known an imaging apparatus provided with an image processing means that performs an automatic motion compensation and corrects a subject shift between images (see, for example, Patent Document 2).

  In addition, the image pickup apparatus includes a camera shake correction processing unit based on the same image processing, and performs motion prediction and motion detection based on a motion vector of continuous inter-field images generated by camera shake, and extracts an image clipping position (from an image memory). There is also known one that corrects camera shake by changing the reading position (see, for example, Non-Patent Document 1).

Japanese Patent Laid-Open No. 7-98466 (page 1-8, FIG. 1) JP 2001-86398 A (page 1-9, FIG. 1) Hiroya Kusaka, Yasunori Kishi, “High Resolution Pure Electronic Image Stabilization System”, Journal of the Institute of Image Information and Television Engineers, 2002, Vol. 56, no. 10, p. 1663-1668

  However, the conventional imaging device described in Patent Document 1 requires a mechanical component for canceling vibration caused by camera shake, and also requires an actuator and a motor for driving the device. There is a problem that power consumption and cost increase as the size increases. Furthermore, since the mechanical parts are affected by the drop impact, it is necessary to design for impact.

  Further, in another conventional imaging device described in Patent Document 2, a plurality of captured images obtained by giving a plurality of consecutive exposures are added, and thus all of the plurality of captured images are stored. A separate frame memory is required.

  Furthermore, as described in Non-Patent Document 1, when camera shake correction is performed based on the motion vectors of continuous inter-field images, it is effective for moving images, but occurs in one image. It cannot be used to correct camera shake on a still image. Further, since an image is cut out from the image memory, a necessary memory capacity is also increased.

  Note that image blur caused by reasons other than camera shake, such as image blur caused by relative movement of the imaging device with respect to the subject when the imaging device is mounted on a moving body such as a vehicle and imaging is performed while moving, is performed. There is a similar problem.

  The present invention has been made to solve the above-described problems, and does not require a special mechanism component or a large-capacity memory, and is small and low-cost by performing blur correction in a solid-state imaging device. An object is to provide an imaging device.

The present invention
An imaging device including a photoelectric conversion unit that receives light from a subject and performs photoelectric conversion ,
Reading means for reading an image charge accumulated in the photoelectric conversion unit, N times during the charge accumulation time (N is an integer of 2 or more) is divided into,
Image holding means for temporarily holding the image charges read by the reading means;
Image shifting means for shifting the image charge held in the image holding means;
An image sensor comprising:
Motion detection means for detecting the motion of the image sensor;
The image charge read out for the first time by the reading means and held in the image holding means is shifted by a first movement amount corresponding to the amount of movement detected by the motion detecting means, and the shifted image charge is shifted. to the image charge, the second time to read image charges by the readings out means to perform the process of adding in the image holding unit,
When N is larger than 2, the image charge obtained as a result of addition of the image charges read out n-th time (n is an integer from 2 to (N−1)) by the reading unit, and the n-th shift movement amount of corresponding to the movement amount detected by the movement detector, in the shifted image charges, said first (n + 1) th to the read image charges by the readings out means Control means for repeatedly performing the process of adding by the image holding means from 2 to (N-1) in order,
The control means provides an imaging device characterized in that the number of times image charge is read is increased so that the amount of movement of image charge by the image charge shift means does not exceed a predetermined upper limit value.

  According to the present invention, it is possible to provide a small-sized and low-cost imaging device by performing blur correction in a solid-state imaging device without requiring a special mechanism device or memory.

  Hereinafter, the case where the image blur is caused by camera shake will be described. However, the present invention is also applicable to image blur that occurs for reasons other than camera shake. For example, the present invention can be applied to a camera that is mounted on an automobile or other moving body and performs shooting while moving and the direction of the main movement is known in advance.

Embodiment 1 FIG.
FIG. 1 is a diagram showing a basic configuration of an imaging apparatus according to Embodiment 1 of the present invention. As shown in the figure, the imaging apparatus includes a lens 1, a solid-state imaging device 2, an analog signal processing unit 3, a pixel signal A / D conversion unit 4, a signal processing unit 5, a display unit 6, a CPU 7, The timing generator 8, the camera shake sensor 9, the motion signal A / D conversion unit 10, the integrating unit 11, and the lens driving unit 12 are included.

  A subject image incident through the lens 1 is photoelectrically converted by the solid-state imaging device 2. This lens 1 functions as a zoom lens having a variable focal length. By the control signal from the lens driving unit 12, the lens 1 can move back and forth on the optical axis, and the focal length can be continuously changed. This control signal is output from the CPU 7.

As shown in FIG. 2, the solid-state imaging device 2 includes a photoelectric conversion unit 19 having a plurality of photoelectric conversion element arrays 20 and a plurality of vertical transfer units 22 provided adjacent to each other corresponding to the photoelectric conversion element arrays 20. A readout gate unit 24 positioned between the photoelectric conversion element array 20 and the vertical transfer unit 22, a horizontal transfer unit 27, and an output amplifier 28.
Each photoelectric conversion element array 20 is arranged at a plurality of pixel positions in a first direction on the imaging screen, for example, the vertical direction, and receives light from a subject (not shown) via the lens 1 to perform photoelectric conversion. It has a plurality of photoelectric conversion elements 21 for performing.
Each vertical transfer unit 22 is provided corresponding to each photoelectric conversion element 21 of the corresponding photoelectric conversion element array 20, and the electric charge accumulated in the corresponding photoelectric conversion element 21 is read by each read gate 25 of the read gate unit 24. And a plurality of transfer elements 23 that transfer charges between each other in the vertical direction.
The horizontal transfer unit 27 receives the electric charge transferred from the vertical transfer unit 22 and output from the vertical transfer unit 22, and transfers the charge in a second direction orthogonal to the first direction, for example, the horizontal direction.
The output amplifier 28 amplifies the charge transferred from the horizontal transfer unit 27 and output from the horizontal transfer unit 27 and outputs the amplified charge.

  The photoelectric conversion in the photoelectric conversion element 21 of the photoelectric conversion unit 19 accumulates an amount of charge corresponding to the intensity of light, and the read pulse TG extracts the charge to the vertical transfer unit 22 via the read gate unit 24. The signals are sequentially transferred in the direction of the horizontal transfer unit 27 by a vertical transfer pulse VDP including four-phase pulses φV1, φV2, φV3, and φV4. The horizontal transfer unit 27 transfers charges in the horizontal direction by a horizontal transfer pulse HDP composed of two-phase pulses φH1 and φH2 every time a pixel for one line enters, and converts the signal into a signal voltage by an output amplifier 28 to output a signal. To do.

  In the present application, for simplicity, the transfer in the vertical transfer unit 22 may be simply referred to as “vertical transfer”, and the transfer in the horizontal transfer unit 27 may be simply referred to as “horizontal transfer”.

With the above-described vertical transfer pulse VDP, charges in the vertical transfer unit 22 are transferred in the direction of the horizontal transfer unit 27 (the direction of the arrow FW in the vertical transfer unit 22 in FIG. 2, hereinafter referred to as “forward direction”). It is also possible to transfer in a direction away from the horizontal transfer unit 27 (a direction opposite to the arrow FW in the vertical transfer unit 22 in FIG. 2, hereinafter referred to as “reverse direction”). When the vertical transfer pulse VDP is not applied, the vertical transfer unit 22 functions as an image holding unit, and charges taken out to the vertical transfer unit 22 through the read gate unit 24 are transferred to the vertical transfer unit 22. Thus, it can be held in the taken-out position without being moved in either the forward direction or the reverse direction.
Each transfer element 23 of the vertical transfer unit 22 also functions as an image addition unit or an image synthesis unit that adds the charge held inside and the charge transferred from the corresponding photoelectric conversion element 21.

  FIG. 3 is an enlarged view showing four pixels arranged in the vertical direction in FIG. A symbol PX indicates one pixel. Each pixel includes a photoelectric conversion element 21, a read gate 25, and a transfer element 23. In the present embodiment, a four-phase vertical transfer pulse VDP is used, and there are four electrodes 26a to 26d per pixel. The electric charge photoelectrically converted by the photoelectric conversion element 21 is read out to the transfer element 23 via the read gate 25. As described below, by changing the voltage of the four-phase vertical transfer pulse VDP in a predetermined time series, the charge in the vertical transfer unit 22 is transferred in the forward direction and the reverse direction by the desired number of pixels. can do.

  The photoelectric conversion elements 21 of the photoelectric conversion unit 19 of the solid-state imaging element are covered with color filters of different colors, and the color filters are periodically arranged in a predetermined repeating pattern. This periodic arrangement may be a Bayer type as shown in FIG. The color filters are, for example, red (R), green (G), and blue (B) color filters. From the photoelectric conversion elements covered with the R, G, and B color filters, R, G, and B pixels are used. A signal is obtained. In FIG. 4, symbol H indicates the horizontal axis on the imaging screen, that is, the horizontal axis, 1, 2,... M along the horizontal axis indicate coordinate values on the horizontal axis, and symbol V indicates the imaging screen. The upper vertical axis, that is, the vertical axis is shown, and 1, 2,..., N along the vertical axis represent coordinate values on the vertical axis.

  In addition, instead of the Bayer array shown in FIG. 4, the honeycomb array shown in FIG. 5 may be used. The honeycomb array is obtained by rotating the Bayer array by 45 degrees, and in the honeycomb array, the imaging elements each include photoelectric conversion elements of pixels that are substantially regular octagons, and the vertical transfer unit 22 extends along the pixel columns ( The shape meanders between adjacent pixel columns.

Referring to FIG. 1 again, output signals (pixel signals) of R, G, and B obtained from the solid-state imaging device 2 are subjected to correlated double sampling processing (CDS) and automatic gain control amplification processing (analog processing) in the analog signal processing unit 3. AGC).
A readout pulse TG, a vertical transfer pulse VDP, a horizontal transfer pulse HDP, and an overflow drain pulse OFD (to be described later) supplied to the solid-state imaging device 2 (collectively referred to as “a solid-state imaging device driving pulse”). The sampling pulse DSP for CDS supplied to the analog signal processing unit 3 is supplied by the timing generator 8. The generation timing of these pulses is controlled by the CPU 7. The AGC in the analog signal processing unit 3 is also performed by a control signal from the CPU 7.

  The output signal of the analog signal processing unit 3 is converted into a digital signal by the pixel signal A / D conversion unit 4 and then processed by the signal processing unit 5 to become a video signal that can be displayed on the display means 6. . The signal processing unit 5 performs image processing such as conversion from RGB signals to luminance color difference signals (YCbCr), white balance correction, γ (gamma) correction, color interpolation processing, and contour enhancement correction. The display means 6 composed of an LCD or the like displays the video signal output from the signal processing unit 5. When shooting a still image, the photographer determines the composition while viewing the image of the subject on the display means 6.

  The camera shake sensor 9 is used as a motion detection unit that senses the movement of the imaging apparatus body, particularly the solid-state imaging device 2 in the first direction (vertical direction) when the photographer images a subject. Here, “sense the movement in the first direction” may be the one that senses only the movement in the first direction, and senses the movement in the other direction. It is also possible to extract a signal representing the motion component of The camera shake sensor 9 is, for example, an element that detects an angular velocity, and a displacement amount and a displacement direction due to vibration can be obtained by time-integrating the obtained angular velocity. The angular velocity signal detected by the camera shake sensor 9 is converted into a digital signal by the motion signal A / D converter 10 and then output to the CPU 7. The camera shake sensor 9 is not limited to an angular velocity sensor, and instead, an angular acceleration sensor, an acceleration sensor, a geomagnetic sensor, or the like can be used, and by performing appropriate signal processing according to each sensor signal, The same effect as when the angular velocity sensor is used can be obtained.

The CPU 7 performs overall control of the imaging system and functions as means for performing various calculations. Based on the video signal data from the signal processing unit 5, automatic exposure control (AE) / automatic white balance control (AWB) is performed, amplification gain setting for the analog signal processing unit 3, and solid-state imaging generated by the timing generator 8 Control of generation of driving pulses for the element 2 and generation of control signals for the lens driving unit 12 are performed.
The integrating unit 11 integrates video signals for one screen input from the signal processing unit 5. The integration result represents the brightness of the captured image. Based on this, the CPU 7 controls the generation of the overflow drain pulse OFD, thereby controlling the charge accumulation time.

  Further, the CPU 7 integrates the angular velocity signal converted into the digital signal by the motion signal A / D conversion unit 10 to calculate the angle and direction of the shake generated in the imaging apparatus, and further, the amount of camera shake based on the calculated value. It also has a function of converting to the number of pixels of image blur.

  Further, as will be described in detail later, the waveform of the vertical transfer pulse VDP for moving charges in the vertical transfer unit 22 (see FIG. 2) and the generation timing of the vertical transfer pulse VDP are determined. A control signal is output to the timing generator 8 based on the determined waveform and generation timing of the vertical transfer pulse VDP, and the timing generator 8 generates a driving pulse signal for the solid-state imaging device 2 based on the control signal. . As described above, the timing generator 8 that has received the control signal generated by the CPU 7 functions as a vertical transfer signal generating means for moving charges in the vertical transfer unit 22 described later.

  Further, the CPU 7 and the timing generator 8 control the timing of the readout pulse for transfer (reading) from the photoelectric conversion element to the transfer element when motion is detected by the camera shake sensor 9, and during a predetermined frame period. In addition, control for reading the image a plurality of times is also performed.

FIG. 6 is a functional block diagram showing components for performing camera shake correction in the present embodiment. The light image incident through the lens 1 is photoelectrically converted by the charge accumulating means 31, and charges are accumulated.
In this embodiment, the charge storage unit 31 is configured by the photoelectric conversion unit 19 in FIG. The motion detection means 32 is a means for detecting the motion of the imaging apparatus, and is configured by the hand movement sensor 9 of FIG.
The synthesizing unit 33 generates an optical image formed on the solid-state imaging device 2 via the lens 1 when the amount of motion (BL) per predetermined time detected by the motion detecting unit 32 is a predetermined value or more. Corresponding first to Nth image signals (N is an integer of 2 or more) in a predetermined frame period are read, and the read first to Nth image signals are detected as motions, respectively. Is shifted by the number of pixels corresponding to the amount of the image.
As shown in the figure, the synthesizing unit 33 includes a reading unit 34, an image holding unit 35, and an image shifting unit 36.
In the first embodiment, the image reading unit 34 is configured by the reading gate unit 24, the image holding unit 35 and the image shift unit 36 are both configured by the vertical transfer unit 22, and the control unit 37 is configured by the CPU 7 and the timing generator 8. Composed.

The reading means 34 reads the image charges accumulated in the charge accumulating means 31 in N times (N is an integer of 2 or more) during the charge accumulation time.
The image holding unit 35 temporarily holds the image charge read by the reading unit 34.
The image shift unit 36 shifts the image charge held by the image holding unit 35.
The image holding unit 35 described above also has a function of adding the shifted image charge and the image charge read by the reading unit 34.

The control unit 37 shifts the image charge read out for the first time and held in the image holding unit 35 by the first movement amount corresponding to the amount of movement detected by the movement detection unit 32, and is shifted. The image holding means 35 adds the image charge read out second by the image reading means 34 to the image charge. The amount of movement is represented by the number of pixels, for example.
When N is larger than 2, the control unit 37 further calculates the image charge obtained as a result of the addition of the image charges read out n-th time (n is an integer from 2 to (N−1)). The image charge shifted by the nth movement amount detected by the motion detection means 32, and the image charge read by the image reading means 34 for the (n + 1) th time is used as the image charge means 35. The process of adding is repeated in order from 2 to (N−1). That is, when N is greater than 2, the control unit 37 uses the reading unit 34 to read the image charge that has been read for the first time and is held by the image holding unit 35. The image charge is shifted by the first movement amount according to the image charge, and the image charge read by the image reading unit 34 for the second time is added to the shifted image charge (the first read and shifted image charge). Are added by the image holding means 35, and the image charge obtained as a result of the addition of the image charge read at the second time and the image charge read at the first time are similarly detected by the motion detection means 32. The image moving unit 35 shifts the image charge read out by the image reading unit 34 for the third time to the shifted image charge by the image holding unit 35. Nth Read, until the addition is completed, the same process is repeated.

  For example, a period from the start of charge accumulation to the first reading (this period may be referred to as a “first image generation period”) and a period from each reading to the next reading (the n-th reading). The period from the readout to the (n + 1) th readout may be referred to as the “nth image generation period”), and the amount of motion corresponds to the amount of motion. The first movement amount and the nth movement amount, which are represented by the number of pixels of the element and correspond to the amount of movement, are set to 1 / N of the amount of movement.

  The solid-state imaging device 2 according to the present embodiment is, for example, an interline type CCD imaging device, and corresponds to an “all pixel readout” method in which all pixel signals are read out sequentially in a horizontal direction line by line sequential scanning. ing.

  FIG. 7 is a schematic diagram showing the operation of the synthesizing means 33 over time. Add to the image signal representing the image generated in each of the first to Nth image generation periods obtained by dividing the charge accumulation period of the predetermined frame period by N for the purpose of camera shake correction in this embodiment. This shows the operation of synthesizing. The first image generation period GC1 indicates a period during which charge is accumulated in the photoelectric conversion unit 19 in order to obtain the first image (first image). Hereinafter, the second image generation period GC2 to Nth The image generation period GCN has the same meaning. After the image signal representing the first image formed by the charge accumulation in the first image generation period GC1 is read (RD1) to the vertical transfer unit 22, the photoelectric conversion unit obtains the image signal representing the second image During charge accumulation (GC2) at 19, the image signal representing the first image is shifted by the image shift means 36 by the number of pixels corresponding to 1 / N of the amount of movement (SH1), and the second image is read out. Until the pixel position after the shift is held by the image holding means 35 (HL1).

  Thereafter, the image signal held at the pixel position by the image holding means 35 is added by the image adding means 34 to the image signal representing the second image (AD1), and the added image is obtained by the image shifting means 36. The pixel is shifted by the number of pixels corresponding to 1 / N of the amount of movement (SH2), and held at the pixel position after the shift by the image holding unit 35 until the third image is read (HL2). Thereafter, the image signal held at the pixel position by the image holding means 35 is added by the image adding means 34 to the image signal representing the third image (AD2). Thereafter, the same operation is repeated until the image signals representing the Nth image are added to obtain a synthesized image signal.

  FIGS. 8 and 9 are timing charts of potential change of the vertical transfer pulse VDP, as a specific method for transferring the charge in the vertical transfer unit 22 of the solid-state imaging device 2 in the forward direction shown in FIGS. Will be described. FIG. 8 is a timing chart of the four-phase pulse φVi (i = 1, 2, 3, 4) of the vertical transfer pulse VDP of each electrode, with the horizontal axis representing time and the vertical axis representing four electrodes 26a to 26d. The applied potentials are shown side by side. FIG. 9 is a diagram showing the state of charge transfer when the vertical transfer pulse VDP shown in FIG. 8 is applied to each electrode. In FIG. 9, the horizontal axis indicates the positions of the electrodes 26 a to 26 d of the transfer element 23 of the vertical transfer unit 22, and the vertical axis indicates the time-dependent change of the potential energy φi (= −eφVi) and the charge QC distribution of each electrode. (Charge distributions at different times t = a1 to a9) are shown.

  Next, the operation will be described. In FIG. 8, since the potential φV2 of the electrode 26b and the potential φV3 of the electrode 26c are + V and the potential φV1 of the electrode 26a and the potential φV4 of the electrode 26d are 0 V at time a1, as shown in FIG. A potential well is formed in the electrode 26c, and charge QC is accumulated. Next, when a bias voltage + V is applied to the electrode 26d at time a2, the charge QC that has been in the electrodes 26b and 26c is diffused from the electrode 26b to the electrode 26d. Next, when the electrode 26b is unbiased at time a3, the charge QC converges from the electrode 26c to the electrode 26d, and as a result, from time a1 to a3, the charge QC is one electrode in the forward direction (rightward in FIG. 9). It ’s moved by the minute.

  By applying a bias voltage + V to the electrode 26a at time a4 and making the electrode 26c non-biased at time a5, the charge QC further moves forward by one electrode. By applying a bias voltage + V to the electrode 26b at time a6 and making the electrode 26d non-biased at time a7, the charge QC further moves forward by one electrode. At time a8, a bias voltage + V is applied to the electrode 26c, and at time a9, the electrode 26a is made non-biased, so that the state becomes the same as at time a1. That is, from time a1 to time a9, the charge QC for one pixel is moved in the forward direction.

  On the other hand, a case where the charge QC in the vertical transfer unit is transferred in the reverse direction will be described with reference to FIGS. 8 used for the description of the case where the charge QC is transferred in the forward direction corresponds to FIG. 10, and FIG. 9 corresponds to FIG. Times a1 to a9 correspond to b1 to b9, respectively.

  Next, the operation will be described. In FIG. 10, at time b1, the bias voltage + V is applied to the electrode 26b and the electrode 26c, and the electrode 26a and the electrode 26d are 0 V with no bias. Therefore, as shown in FIG. 11, potential wells are formed in the electrode 26b and the electrode 26c. The charge QC is accumulated. Next, when a bias voltage + V is applied to the electrode 26a at time b2, the charge QC that was in the electrodes 26b and 26c diffuses from the electrode 26a to the electrode 26c. Next, when the electrode 26c is unbiased at time b3, the charge QC converges from the electrode 26a to the electrode 26b, and as a result, the charge QC is one electrode in the reverse direction (leftward in FIG. 11) from time b1 to b3. It ’s moved by the minute.

  By applying a bias voltage + V to the electrode 26d at time b4 and making the electrode 26b non-biased at time b5, the charge QC further moves in the reverse direction by one electrode. By applying a bias voltage + V to the electrode 26c at time b6 and making the electrode 26a unbiased at time b7, the charge QC further moves in the reverse direction by one electrode. At time b8, a bias voltage + V is applied to the electrode 26b, and at time b9, the electrode 26d is made non-biased, so that the state is the same as at time b1. That is, the charge for one pixel is moved in the reverse direction from time b1 to time b9.

  In the present embodiment, as described above, the convergence step is performed after the charge diffusion step and the charge transfer is realized by repeating this. However, the charge transfer is performed even if the diffusion and convergence steps are performed simultaneously. Is possible. For example, similar charge transfer can be achieved by simultaneously performing diffusion at time a2 and convergence at time a3 in FIG. 9, that is, by simultaneously applying a bias voltage to the electrode 26d and making the electrode 26b non-biased. Alternatively, even if convergence at time a3 and diffusion at time a4 are performed at the same time in FIG. 9, charge transfer is possible.

  Furthermore, in this embodiment, charges are transferred by a four-phase vertical transfer pulse VDP. However, the transfer is not limited to four phases, and charges can be transferred by the same method as long as there are three or more phases. Similarly, the horizontal transfer pulse HDP is a two-phase one, but is not limited thereto.

  In the present embodiment, a CCD image sensor is used as the image sensor. However, the present invention is not limited to this. For example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor including means equivalent to the above-described combining means 33 is used. It doesn't matter.

  Next, the concept of “camera shake” to be corrected in the present invention will be described. FIG. 12A is a diagram illustrating a relationship between a camera shake direction and a captured image when the camera operator shakes a camera shake in the vertical (pitching) direction when imaging a subject. Here, the subject is a circular light source LS, and a captured image when captured without causing camera shake is illustrated in FIG. An image LSb holding a round subject shape is captured.

  The photographer HY first holds the imaging device PG so that the optical axis coincides with the ground GL and the line HL, and presses the shutter button SB. Camera shake occurs at this time, and when the optical axis OX of the imaging device PG moves upward ((+) direction) due to camera shake, the captured image La is represented by the symbol TR as shown in FIG. As shown in FIG. 12D, when tailing occurs in the downward direction and the optical axis OX of the imaging device PG moves in the downward direction ((−) direction), the captured image Lc has As indicated by the symbol TRc, tailing occurs in the upward direction.

  An object of the present invention is to realize a high-quality captured image in which the tails TRa and TRb of the subject images LSa and LSc reproduced in FIGS. 12B and 12D are reduced. In most cases, the image quality degradation due to camera shake is caused by rotation of the imaging device or the like caused by the operation of the photographer HY pressing the shutter button SB. Since this rotation direction substantially coincides with the pressing direction of the shutter button SB, a practically sufficient correction is possible by one-dimensional correction corresponding to the pressing direction. In the present embodiment, the direction in which charges are transferred in the vertical transfer unit 22 (vertical direction on the imaging screen) corresponds to the direction in which camera shake is likely to occur (when the imaging screen is provided vertically). In the vertical transfer unit 22, the direction in which charges are transferred (the vertical direction on the imaging screen) is made substantially parallel to the direction in which camera shake is likely to occur.

FIGS. 13A to 13G, FIGS. 14A to 14G, and FIGS. 15A to 15G are timing charts showing the operation of the imaging apparatus according to the present embodiment.
FIGS. 13 (a), 14 (a), and 15 (a) show the shutter signal BP, and a transition from the high level to the low level occurs at the time Ton when the pressing of the shutter is detected.
FIGS. 13B, 14B, and 15B are vertical synchronization signals VD, which are generated in a predetermined cycle (vertical cycle) in principle. However, the charge accumulation time determined by the AE control by the CPU 7 is shown. As a result, the vertical synchronization signal VD is suppressed, and a frame period FP (3) having a length longer than usual, which is four times as long as usual in the illustrated example, is formed.

  The overflow drain pulses in FIGS. 13 (c), 14 (c) and 15 (c) are applied to the overflow drain electrode of the solid-state imaging device 2 from the timing generator 8 in synchronization with the horizontal synchronization signal. The electronic shutter function is realized by discharging the charge accumulated in the photoelectric conversion element 21 onto the substrate in the solid-state imaging element 2 in order to adjust the charge accumulation period in the control. Of the overflow drain pulse OFD in FIG. 13C, pulses other than the first pulse OFD1 (generated simultaneously with or immediately after the rising of the vertical synchronization signal VD (first after the rising)) Sometimes called a shutter pulse.

  13D, 14D, and 15D show a period PD in which photoelectric conversion and charge accumulation are performed in the photoelectric conversion element 21. FIG. This period PD (indicated by symbols SS1, SS2, etc., respectively) starts after the generation of the overflow drain pulse OFD in each frame period and continues until the next rising edge of the vertical synchronization signal VD. In other words, the time point at which the generation of the column of the overflow drain pulse OFD is finished is determined so that the charge accumulation period has a given length.

FIGS. 13E, 14E, and 15E are readout pulses TG for reading out (transferring) the charge accumulated in the photoelectric conversion element 21 to the corresponding transfer element 23 of the vertical transfer unit 22. The read gate 25 is controlled.
FIGS. 13 (f), 14 (f) and 15 (f) show vertical transfer pulses VDP (φV1 to φV4), and FIG. 13 (g) shows horizontal transfer pulses HDP (φH1, φH2).

  Prior to pressing the shutter button SB, a charge accumulation period SSi (i = 1, 2,...) Defined as one of the exposure conditions is set based on the brightness of the subject and imaging is performed. For example, the charge accumulated in the charge accumulation period SS1 in the frame (first frame FP (1)) starting in the first vertical blanking period VB (1) in which a certain vertical synchronization signal VD is generated is the first This is read to the vertical transfer unit 22 by the read pulse TG10 generated in the first vertical blanking period VB (2) of the frame following the frame FP (1) (second frame FP (2)). In the period 100b in the second frame FP (2), the vertical transfer unit 22 transfers the frame to the horizontal transfer unit 27. Transfer to the horizontal transfer unit 27 is simultaneously performed in the plurality of vertical transfer units 22. The pixel signals transferred at the same time are a plurality of pixels aligned in the horizontal direction on the imaging surface, that is, signal charges read from the plurality of photoelectric conversion elements 21 respectively constituting the plurality of pixels on the same line, Are transferred by the vertical transfer unit 22. In other words, the signal charges transferred from the respective photoelectric conversion element arrays 21 of the photoelectric conversion unit 20 to the vertical transfer unit 22 are sequentially transferred to the horizontal transfer unit 27 line by line. Then, in the period 101b in the same second frame period FP (2), the charges transferred from the vertical transfer unit 22 to the horizontal transfer unit 27 are transferred to the output amplifier 28 in the horizontal transfer unit 27 and output. Amplified by the amplifier 28 and output.

  If the shutter button SB is pressed at a certain time Ton in the charge accumulation period SS2 in the second frame period FP (2) and the shutter signal BP transitions from a high level to a low level, the second frame FP (2 The charge accumulated in the charge accumulation period SS2 is applied with the overflow drain pulse OFD1 to the solid-state imaging device 2 at the beginning of the next vertical blanking period VB (3) (substantially simultaneously with the rise of the vertical synchronization signal VD). As a result, the sheet is discharged onto the substrate without being transferred to the vertical transfer unit 22. Note that the overflow drain pulse OFD1 is not necessarily required when an electronic shutter pulse (overflow drain pulse OFD other than OFD1) is generated by AE control within the vertical blanking period. If the generation of the readout pulse (such as TG10 in VB (2)) is prohibited, the reading of the signal charge to the vertical transfer unit 22 is prohibited, so that it is accumulated when the electronic shutter pulse is generated by the AE control. This is because the charged charges are discharged onto the substrate.

The frame period FP (3) next to the time point Ton when the pressing of the shutter button SB is detected is a period in which charge accumulation is performed to obtain a video signal for an image to be recorded as a still image. In this frame period FP (3), the charge accumulation period is divided into a plurality (N) of image generation periods (see FIG. 7), and the charge is read out to the vertical transfer unit 22 for each image generation period. Every N) images.
FIG. 13, FIG. 14 and FIG. 15 show cases where the number of divisions (and hence the number of readings) N of the charge accumulation period is 2, 4, and 8, respectively.

  The plurality of images are finally combined into one image and output from the solid-state imaging device. Until then, among the images constituting the plurality of images, the images read during the first image generation period ( An image read out during the second image generation period (second image) after temporarily holding in the vertical transfer unit 22 (which may be referred to as “first image” hereinafter). Is moved (transferred) by the image position obtained by dividing the amount of motion detected by the motion detection unit 32 by the number of the plurality of images before the image is read to the vertical transfer unit 22, and then the image is transferred to the image. After being added to the image (second image) read out during the second image generation period, it is temporarily stored. Thereafter, the same processing is repeated until the last read image (Nth image) is added.

  The number N of readings is determined by the control means 37 (comprising the CPU 7 and the timing generator 8). The control unit 37 determines the number of times of reading N according to the amount of motion detected by the motion detection unit 32, and the amount of movement of image charge by the image charge shift unit 36 (vertical transfer unit 22) is predetermined. The number of readings N is set so as not to exceed the upper limit value Ln. That is, when the number of times of reading N is fixed, it is necessary to increase the amount of movement as the amount of motion detected by the motion detector 32 increases. However, as the amount of movement increases, the number of times of reading N increases. The movement amount is made not to exceed the upper limit value Ln.

  When the number of readings (the number of divisions of the charge accumulation period) is set to 2, as shown in FIG. 13D, the image generation period (after the division) after dividing one charge accumulation period equally. The charge accumulation periods are S1 and S2, respectively. In this case, the charges accumulated in the image generation period S1 are read out to the vertical transfer unit 22 by the readout pulse TG1, and the charges accumulated in the image generation period S2 are read out by the readout pulse TG2 (FIG. 13 (e)). Reference numeral 102a in FIG. 13F denotes a period in which the charges read by the read pulse TG1 are transferred by a predetermined number of pixels in the forward transfer unit 22 in the forward or backward direction.

  When the number of readings (the number of divisions of the charge accumulation period) is set to 4, as shown in FIG. 14D, the image generation period (after the division) after dividing one charge accumulation period equally. The charge accumulation period is S1 to S4. In this case, the charges accumulated in the image generation periods S1 to S4 are read out to the vertical transfer unit 22 by read pulses TG1 to TG4 (FIG. 14E), respectively. Reference numerals 103a to 103c in FIG. 14F denote periods in which charges read by the read pulses TG1 to TG3 are transferred by a predetermined number of pixels in the forward transfer unit 22 in the forward direction or the reverse direction, respectively.

  When the number of readings (the number of divisions of the charge accumulation period) is set to 8, as shown in FIG. 15D, the image generation period (after the division) after dividing one charge accumulation period equally. The charge accumulation period) is S1 to S8. In this case, the charges accumulated in the image generation periods S1 to S8 are read out to the vertical transfer unit 22 by read pulses TG1 to TG8 (FIG. 15E), respectively. Reference numerals 105a to 105g in FIG. 15F denote periods in which charges read by the read pulses TG1 to TG7 are transferred by a predetermined number of pixels in the vertical transfer unit 22 in the forward direction or the reverse direction, respectively.

  When the number of times of reading N is 2, the number of pixels in the vertical transfer unit 22 is proportional to ½ of the amount of motion detected by the motion detector 32, that is, the number of pixels corresponding to ½ of the number of pixels corresponding to the amount of motion. To transfer the charge. For example, when there is a charge transfer period represented by reference numeral 102a as shown in FIG. 13 (f), charge transfer is performed in proportion to 1/2 of the total camera shake amount in the charge transfer period. 2 is corrected. However, half of the total camera shake amount remains as it is, and as a result, camera shake is reduced to ½.

  When the number of times of reading N is 4, the vertical transfer unit 22 is in proportion to ¼ of the amount of motion detected by the motion detector 32, that is, ¼ of the number of pixels corresponding to the amount of motion. To transfer the charge. For example, when there are three charge transfer periods represented by reference numerals 103a to 103c as shown in FIG. 14F, charge transfer is performed in proportion to ¼ of the total camera shake amount in each period. As a total of two charge transfer periods, 3/4 of the total camera shake amount is corrected. However, ¼ of the total camera shake amount remains as it is, and as a result, camera shake is reduced to ¼.

When the number of times of reading N is 8, the vertical transfer unit 22 has an amount proportional to 1/8 of the amount of motion detected by the motion detector 32, that is, the number of pixels corresponding to 1/8 of the number of pixels corresponding to the amount of motion. To transfer the charge. For example, when there are seven charge transfer periods denoted by reference numerals 105a to 105g as shown in FIG. 15 (f), charge transfer is performed in proportion to 1/8 of the total camera shake amount in each period. 7/8 of the total camera shake amount is corrected as the total charge transfer period. However, 1/8 of the total camera shake amount remains as it is, and as a result, camera shake is reduced to 1/8.
Generally, if the number of divisions in the charge accumulation period is N, camera shake can be reduced to 1 / N.

Normally, in order to reduce image blur due to camera shake of a recorded image to ½ without using camera shake correction as in the present embodiment, the exposure aperture is opened by one step (F value is set to a value larger by one step). In other words, this can be achieved by doubling the amount of light taken into the solid-state imaging device and halving the charge accumulation period (shutter speed). In this embodiment, however, the exposure aperture is not opened one step. Can reduce camera shake by half.
In general, when the number of divisions in the charge accumulation period is N, it is equivalent to increasing the aperture of the exposure by log ₂ N stages by reading out images N times and synthesizing N images. A correction effect can be expected. Furthermore, the number of divisions in the charge accumulation period is limited to a value given by 2 to the power of M (M is an integer of 1 or more), and by increasing M by 1, the number of divisions is increased and M is decreased by 1. Thus, the number of divisions may be reduced.

  When there is no camera shake (below a predetermined threshold value), it is not necessary to perform vertical transfer every time the image generation period S ends, and therefore the vertical transfer pulse VDP is not applied, and FIG. 14 (f) and the periods 102a, 103a to 103c and 105a to 105g for vertical transfer shown in FIG. 15 (f) do not exist. In this case, for example, the read N images are not moved, but are added in the read state to finally obtain one image.

  13 (g), FIG. 14 (g), and FIG. 15 (g), reference numeral 110 denotes the charge in the vertical transfer unit 22 in the horizontal transfer unit 27 direction in the vertical transfer periods 102a, 103a to 103c, and 105a to 105g. After being pushed out to the horizontal transfer unit 27 by transferring to the horizontal transfer unit 27, the excess charge accumulated in the horizontal transfer unit 27 is swept away, that is, the accumulated charge in the horizontal transfer unit 27 (prior to the next vertical transfer). This is a period for zeroing.

In FIGS. 13, 14, and 15, the read pulse TG is applied to the read gate 25 during the vertical blanking period VB (3), so that all charges accumulated in the charge accumulation period SS 2 are transferred to the vertical transfer unit 22. After the reading, the high-speed transfer to the horizontal transfer unit 27 and the high-speed transfer by the horizontal transfer unit 27 are performed to sweep away, thereby making the overflow drain pulse OFD1 unnecessary.
In addition, if the first line for the charge transfer in the horizontal transfer period 101c in the frame period FP (4) is discarded, the horizontal transfer period 110 in the blanking period VB (4) is not necessary. Can do.
Further, in this embodiment, the charge accumulation period is adjusted by the electronic shutter pulse. However, by providing a mechanical shutter, the charge accumulation can be performed only while the shutter is open.

  As described above, when the number of divisions in the charge accumulation period is increased with the increase in the amount of motion detected by the motion detection means 32, the transfer amount (image charge movement amount) in the vertical transfer unit becomes excessively large. Can be prevented, and the influence of camera shake can be reduced.

When the number of readings (the number of divisions of the charge accumulation period) is 2, it is proportional to 1/2 of the amount of motion detected by the motion detector 32, that is, by the number of pixels corresponding to 1/2 of the amount of motion. The charge is transferred in the transfer unit 22, but the upper limit Ln is provided for the number of pixels to be transferred, and the number of pixels proportional to ½ of the amount of motion exceeds the upper limit value Ln. The read count is set to a value larger than 2, for example, 4 or 8.
Similarly, when the number of times of reading is 4, when the number of pixels corresponding to ¼ of the motion amount is proportional to ¼ of the motion amount detected by the motion detection unit 32, that is, within the vertical transfer unit 22. Although the charge is transferred, the upper limit Ln is provided for the number of pixels to be transferred, and when the number of pixels proportional to ¼ of the amount of movement exceeds the upper limit value Ln, the number of times of reading is 4 A larger value, for example, 8 is set.

  Thus, by providing the upper limit value Ln for the amount of movement of charges for each image generation period, when the amount of camera shake is large, the number of readings is increased and the amount of movement of charges by the image shift means 36 is reduced. Blur can be corrected with high accuracy.

  FIG. 16 shows the relationship between the amount of camera shake (BL) and the amount of movement. As the amount of camera shake BL increases, the amount of movement changes as indicated by a thick line with an arrow. TG = 2, 4, 8, 16... Are applied to the solid-state imaging device 2 from the timing generator 8 within the VD period in which the camera shake correction control is performed, respectively, and the number of times of reading TG Number of divisions).

When the number of readings TG is fixed, the movement amount increases in proportion to the camera shake amount (expressed by the number of pixels corresponding to the camera shake amount) BL, and when the camera shake amount exceeds a certain limit value, the movement amount also increases to the upper limit value Ln. Over.
For example, when TG = 2 is fixed, when the camera shake amount BL is a certain value (first value) BL1, the movement amount is the upper limit value Ln, and in the range where the camera shake amount BL is larger than the first value BL1, the movement amount is The upper limit Ln is exceeded. When the camera shake amount BL becomes larger than the camera shake amount (first value) BL1 at which the movement amount reaches the upper limit value Ln at TG = 2, the read count TG is set to a larger value, for example, TG = 4. If TG = 4, the movement amount is ½ compared to the case of TG = 2. That is, the movement amount is 1/4 of the camera shake amount BL.

Similarly, for example, when TG = 4 is fixed, when the camera shake amount BL is the second value BL2, the movement amount is the upper limit value Ln, and in the range where the camera shake amount BL is larger than the second value BL2, the movement amount is the upper limit. The value Ln is exceeded. When the camera shake amount BL becomes larger than the camera shake amount (second value) BL2 at which the movement amount reaches the upper limit value Ln when TG = 4, the read count TG is set to a larger value, for example, TG = 8. If TG = 8, the amount of movement is ½ compared to the case of TG = 4. That is, the movement amount is 1/8 of the camera shake amount BL.
Hereinafter, similarly, (when TG> 2), the number of times of reading is set so that the movement amount does not exceed the upper limit value Ln and ½ of the movement amount does not become smaller than ½ of the upper limit value Ln. Define TG.

  In the above example, the method of converting the camera shake amount BL into the number of pixels from the integrated value of the position of the lens 1 and the angular velocity (obtaining the amount of camera shake represented by the number of pixels) is a variable of the integrated value of the position of the lens 1 and the angular velocity. The table may be used, or may be calculated from a formula given in advance. For example, in FIG. 13 where the number of image generation periods is 4, when the camera shake amount is BL = 16, charges are transferred by BL / 4 = 4 pixels in the vertical transfer periods 103a, 103b, and 103c, respectively.

  FIGS. 17A to 17I show details of a captured image when camera shake occurs during imaging and a subject image on the imaging surface of the solid-state imaging device 2. In this example, the camera shake occurs in the upward direction at the moment the photographer presses the shutter button SB of the image pickup device (the image pickup device moves upward, so that the picked-up image gradually moves downward), and the image generation period This shows a case where the number is 4 and the pixel conversion number of camera shake in the vertical direction is 16. FIG. 17A shows a captured image when there is no camera shake. FIG. 17B is a captured image in the case where the charge accumulation period is not divided and the camera shake correction is not performed, as in the case indicated by the symbols SS1 and SS2 in FIG. 13, and the camera shake of 16 pixels in the vertical direction. Has occurred.

  FIGS. 17C to 17F are images obtained by dividing the image of FIG. 17B into four image generation periods, and charge accumulation in the image generation periods S1, S2, S3, and S4, respectively. And shows a captured image obtained when it is assumed that all pixels are read for each image generation period (in practice, an image for each image generation period is not read out from the solid-state imaging device, After being added by the vertical adder, it is read outside). The image generation periods S1 to S4 have the same length, and the brightness of the captured images in FIGS. 17C to 17F is ¼ of the brightness of the captured image in FIG.

In each of FIGS. 17C to 17F, the difference between the dotted line and the solid line represents a shake during each image generation period.
The deviation BL of the subject image for 16 pixels generated in the image of FIG. 17B can be equally distributed to the deviation of 4 pixels (DBL = BL / 4) in FIGS. 17C to 17F. And If it is assumed that the angular velocity of camera shake is constant within the image generation period of the images of FIGS. 17C to 17F, such a treatment can be approximately performed. In the first embodiment,
It sequentially corrects and adds (synthesizes) the shift of 4 pixels existing between images obtained before and after,
That is, the image of FIG. 17C is shifted by a shift DBL of 4 pixels and synthesized with the image of FIG. 17D to form a first synthesized image, and this first synthesized image is further converted to 4 pixels. The shift DBL is shifted by a minute and synthesized with the image of FIG. 17E to form a second composite image. The second composite image is further shifted by a shift DBL of four pixels to change the image of FIG. ) To form a third composite image (final composite image), whereby the subject image is located at substantially the same position as the captured image of FIG. 17 (f) and the subject image shown in FIGS. 17 (c) to 17 (e) after being shifted and synthesized (the same image as superimposed)) as the final captured image, the solid-state imaging device 2 Is output from

  FIGS. 17G to 17J show images when a subject is projected onto the imaging surface of the solid-state imaging device 2 through a lens. The corresponding images in FIGS. 17C to 17F are respectively There is an upside down relationship.

For the subject images in FIGS. 17G to 17J, the tail portion due to camera shake is not drawn for convenience, but actually the tail portion due to camera shake similar to the subject image in FIGS. Exists. If the number of image generation periods is 4, camera shake that is ¼ of the total camera shake amount occurs for each image in each image generation period.
If the number of image generation periods is 8, a camera shake of 1/8 of the total amount of motion occurs in the images in each image generation period.

  As shown in FIG. 12 (a), when the orientation of the imaging device changes upward due to camera shake, the subject on the imaging surface upside down in FIGS. 17 (g), (h), (i), (j). The image moves upward. In FIG. 17 (g), (h), (i), and (j), “*” marks indicated by reference numerals d1, d2, d3, and d4 indicate one pixel at the same location on the subject (person's face). The details of the charge transfer in the vertical transfer unit 22 will be described below with reference to FIGS. 18 and 19 while paying attention to this pixel.

  FIG. 18 is a diagram in which the period in which the charge accumulation period is divided in FIG. 13 is expanded in the time axis direction, and codes TSa to TSk are assigned to each operation in time series. The state of charge transfer in the vertical transfer unit 22 at the time when these operations TSa to TSk are performed will be described with reference to FIGS.

FIGS. 19A to 19H show the charges in the vertical transfer unit 22 when the subject moves upward on the imaging surface of the solid-state imaging device 2, as shown in FIGS. 17G to 17J. It shows a method for correcting camera shake by transfer. Similarly to FIGS. 2 and 3, in FIGS. 19A to 19J, each photoelectric conversion element array 20 includes a plurality of photoelectric conversion elements 21, and the vertical transfer unit 22 includes a plurality of transfer elements 23. Each transfer element 23 corresponds to a pixel and includes four transfer electrodes, but this point is not shown in FIGS. 19A to 19H.
Each of the read gates 25 of the read gate unit 24 is provided between each photoelectric conversion element 21 and the corresponding transfer element 23.
First, as shown in FIG. 19A, in the first image generation period S1 (TSa), the photoelectric conversion is performed by the photoelectric conversion element 21 (1) corresponding to the target pixel (d1 in FIG. 17G). Accumulate d1.

Next, as shown in FIG. 19B, the charge d1 accumulated in the photoelectric conversion element 21 (1) during the image generation period S1 by the readout pulse TG1 is transferred to the corresponding transfer element 23 (1) of the vertical transfer unit 22. Read (TSb).
Immediately after the end of this charge readout, as shown in FIG. 19C, the charge d2 accumulation (TSc) in the photoelectric conversion element 21 (2) of the target pixel is started in the second image generation period S2, and this charge During the accumulation (TSc), the charge read out to the vertical transfer unit 22 by the operation TSb is moved upward by four pixels from the transfer element 23 (1), that is, moved to the transfer element 23 (2) (TSd).
Next, as shown in FIG. 19D, the charge d2 of the photoelectric conversion element 21 (2) (charge d2 accumulated in the photoelectric conversion element 21 (2) during the image generation period S2) is vertically transferred by the readout pulse TG2. The data is read to the corresponding transfer element 23 (2) of the unit 22 and added to the charge d1 moved from the transfer element 23 (1) to the transfer element 23 (2) in the vertical transfer unit 22 (TSe).

Immediately after the completion of the charge readout and addition, as shown in FIG. 19 (e), the charge d3 accumulation (TSf) in the photoelectric conversion element 21 (3) of the target pixel is started in the third image generation period S3. During this charge accumulation (TSf), the charge (d1 + d2) added in the transfer element 23 (2) of the vertical transfer unit 22 in the operation Tse is upward by four pixels from the transfer element 23 (2), that is, the transfer element 23. Move to (3) (TSg).
Next, as shown in FIG. 19F, the charge d3 of the photoelectric conversion element 21 (3) (charge d3 accumulated in the photoelectric conversion element 21 (3) during the image generation period S3) is vertically transferred by the readout pulse TG3. The data is read to the corresponding transfer element 23 (3) of the unit 22 and added to the charge (d1 + d2) transferred from the transfer element 23 (2) to the transfer element 23 (3) in the vertical transfer unit 22 (TSh).

Immediately after the completion of the reading and addition of the charges, as shown in FIG. 19G, the accumulation (TSi) of the charge d4 in the photoelectric conversion element 21 (4) of the target pixel is started in the fourth image generation period S4. During the charge accumulation (TSi), the charge (d1 + d2 + d3) added in the transfer element 23 (3) of the vertical transfer unit 22 in the operation TSh is upward by four pixels from the transfer element 23 (3), that is, the transfer element. 23 (4) (TSj).
Next, as shown in FIG. 19 (h), the charge d4 of the photoelectric conversion element 21 (4) (charge d4 accumulated in the photoelectric conversion element 21 (4) during the image generation period S4) is vertically transferred by the readout pulse TG4. The data is read to the corresponding transfer element 23 (4) of the unit 22 and added to the charge (d1 + d2 + d3) transferred from the transfer element 23 (3) to the transfer element 23 (4) in the vertical transfer unit 22 (TSk).
Thereafter, as indicated by the arrow Ym, the data of all the pixels is transferred to the horizontal transfer unit 27 (100c in FIGS. 13 and 18), and at the same time, the signal charges are transferred to the output amplifier unit 28 in the horizontal transfer unit 27. (101c in FIGS. 13 and 18).
With the above operation, image data with reduced image blur can be obtained.

FIG. 20 is a flowchart relating to the camera shake correction operation according to the first embodiment of the present invention. First, the read count TG is expressed as TG = 2 ^ m (m = 1, 2,...), And the initial value is set to m = 1 (St0). Next, it is monitored whether or not the shutter button SB is pressed (St1), and when the shutter button SB is pressed, the camera shake amount BL is calculated (St2). In the present embodiment, an angular velocity sensor is used as the camera shake sensor 9, and the angular velocity of vibration generated in the imaging device when the photographer presses the shutter button SB is detected, and the CPU 1 determines the zoom magnification of the lens 1. By converting the blur amount BL into the corresponding number of pixels from the integrated value of the position and the angular velocity, the blur amount represented by the number of pixels is obtained.
Next, the process goes to step St3, where it is determined whether A <BL / TG ≦ Ln is satisfied. This determination is to determine the range of movement amount in one charge accumulation period when the total camera shake amount is BL. Ln is an upper limit value of the movement amount and can be set arbitrarily. The fixed value A is a numerical value such that A = 0 when m = 1, and A = Ln / 2 when m ≦ 2.

If the determination result in step St3 is “NO”, m is increased by 1 in step St4, and the process returns to step St3.
If the determination result in step St3 is “YES”, the process proceeds to step St5, and a value is substituted into “m” in the calculation formula TG = 2 ^ m. That is, the number N of times of reading is determined based on the value of m.

After step St5, the number of times of reading (the number of divided charge accumulation periods) N is determined based on the calculated value of TG = 2 ^ m at that time (step St6).
Next, n = 1 is given for the purpose of first reading an image (first image) obtained by charge accumulation in the first image generation period among the first to N image generation periods (step St7). ). Next, the nth image is read (step St8). In the initial state, n = 1 in step St8, so the first image is read out.
Next, after shifting the image by the image position (number of pixels) obtained by dividing the amount of camera shake (number of pixels representing the amount of motion) BL by the number of times of reading N (step St9), the image is held in the vertical transfer unit 22 (St10). . Next, the (n + 1) th image is read out.
In the initial state (that is, when the process of step St11 is first executed), the (n + 1) -th image indicates the second image.

Then, the held nth image and the read out (n + 1) th image are added (step St12). This addition is performed on each transfer element 23 of the vertical transfer unit 22 that holds the nth image (that is, the charge corresponding to the nth image), and each photoelectric conversion element 21 of the photoelectric conversion element array 20. Is performed by reading out the (n + 1) th image (the charge corresponding to the (n + 1) th image).
Next, it is determined whether or not n = N−1 (step St13). If n = N−1 is not satisfied, 1 is added to n (step St14), the process returns to step St9, and the processes from step St9 to step St13 are repeated. This repetition is continued until n = N−1 in Step St13. If n = N−1, all charges in the vertical transfer unit 22 are transferred to the horizontal transfer unit 27 in the next frame period (step St15), and the process ends.

  FIG. 21 is a diagram illustrating the movement amount during each image generation period with respect to the camera shake amount BL when the upper limit value Ln of the movement amount is changed. As the camera shake amount BL increases, the movement amount changes as indicated by a thick line with an arrow. At this time, assuming that the set value of Ln = Ln2 in FIG. 21B is a standard, FIG. 21A sets Ln to a smaller value Ln1 than FIG. 21B, and FIG. 21C shows FIG. ) Shows a case where Ln is set to a larger value Ln3. When compared with the same camera shake amount BL, the number of times of reading TG is larger in FIG. 21A, whereas the number of times of reading TG is smaller in FIG. 21C.

  For example, in the case where the photographer pushes the shutter by hand without fixing the imaging device, or in the situation where camera shake is likely to occur in shooting under low illumination (exposure time becomes long), the amount of camera shake is as shown in FIG. The upper limit value Ln of the movement amount is set to a low value Ln1 so that the number of times of reading TG increases when BL is large. On the other hand, in a fixed point camera where image blurring hardly occurs or in a situation where camera shake is unlikely to occur in shooting under high illuminance, as shown in FIG. 21C, when the camera shake amount BL is small, the number of times of reading N is reduced. The upper limit value Ln of the movement amount is set to a high value Ln3. Such processing is performed by the CPU 7. By switching the upper limit value Ln of the movement amount in this way, a more appropriate correction effect can be expected by performing the optimum camera shake correction according to the environment in which the imaging apparatus is operated.

  As described above, according to the first embodiment, it is possible to provide an imaging device that reduces and corrects camera shake only by charge movement inside the solid-state imaging device without requiring complicated mechanism parts such as an actuator. In addition, even if camera shake is corrected by dividing the charge accumulation period, frame memory or the like is not necessary, and high-precision camera shake correction is possible even when there are structural restrictions such as cameras for portable devices. Is possible.

In the first embodiment, it has been described that the amount of movement can be detected immediately after the detection Ton of pressing the shutter button SB, that is, between the time Ton and the next rise of the VD signal. Hereinafter, a case will be described in which the detection of the amount of motion does not end before the rise of the first VD signal after Ton.
22A to 22I show time charts in that case. In FIG. 22, it is assumed that the number of readings is 4, and this time chart corresponds to FIG. 14, but the vertical synchronization signal VD is not suppressed, and the frame period FP (3) is another frame period. It has shown the case where it has the same length.
22 (a) to (e), (h), and (i) show the same as those shown in FIGS. In FIG. 22A, the transition (Ton) from the high level to the low level of the signal BP due to the detection of the pressing of the shutter button occurs in the first frame period FP (1).
FIG. 22F shows the output GY of the camera shake sensor (angular velocity sensor) 9. In the example shown in the figure, the output GY rises substantially simultaneously with the fall of the signal BP and thereafter maintains a substantially constant value.
FIG. 22 (g) shows the operation timing in the CPU 7, that is, based on the period SGb in which the CPU 7 integrates the output GY (converted by the motion signal A / D converter 10) of the camera shake sensor 9 and the integrated value. A period SKb for calculating the amount of movement is shown.

  In the illustrated example, the amount of movement is detected in the frame period FP (2) next to the frame period FP (1) in which the pressing of the shutter button SB is detected, and the charge is detected in the next frame period FP (3). The accumulation time SS3 is divided into four to perform camera shake correction imaging. This point will be described in more detail below.

The camera shake sensor 9 outputs a signal in accordance with the movement of the imaging apparatus. For example, when the shutter button SB is pressed and the imaging apparatus moves, output is started in accordance with the movement. The start of the output of the camera shake sensor 9 does not always coincide with the detection timing Ton of pressing the shutter button, and may be a little earlier than that, or may be a little later. After the start of the output of the camera shake sensor, the movement of the imaging device usually does not stop immediately, and continues to move for several frames.
In the illustrated example, the output GY of the shake sensor 9 is output from the time when the generation of the column of the overflow drain pulse OFD is finished after the vertical synchronization signal VD of the frame period FP (2) following the detection Ton of the shutter button press is risen. Integration starts. This integration is continued for a time SGb having the same length as the charge accumulation time SS2 determined based on the brightness of the subject. The charge accumulation time SS2 of the second frame FP (2) and the charge accumulation time SS3 (= S1 + S2 + S3 + S4) of the third frame are both the imaging results of the first frame FP (1) or the previous frame. It is determined based on (the result of integration by the integration means 11) and is the same as each other.
When the integration is completed, the amount of movement is calculated by the CPU 7 based on the integration result. The period for this calculation is indicated by the symbol SKb.
In the third frame FP (3), the amount of motion calculated in the second frame FP (2) is used to perform image stabilization imaging by dividing the charge accumulation time SS3 into four parts. Therefore, if the speed of movement of the imaging device does not change between the second frame FP (2) and the third frame FP (3), the amount of movement and the charge accumulation time obtained by integration in the integration time SGb. The amount of motion in SS3 is exactly the same, and correction can be performed accurately. The movement of the imaging device due to camera shake usually continues over several frames, and the difference between the speed of movement in the second frame and the speed of movement in the third frame is small.

  As shown in FIG. 22, after the vertical synchronization signal VD rises in the frame period FP (2) following the detection Ton of the shutter button press, the camera shake sensor starts from the point when the generation of the overflow drain pulse OFD sequence ends. Instead of starting the integration of the output of 9, the integration of the output of the camera shake sensor 9 may be started simultaneously with the rise of the vertical synchronization signal VD in the frame period FP (2) following the detection Ton of pressing the shutter button. . Also, the integration of the output of the camera shake sensor 9 is started simultaneously with the detection Ton of the shutter button press, and the imaging result (the subject brightness, that is, the integration means 11 of the integration unit 11) is started before the frame where the shutter button press is detected. Based on the output, the integration time SGb and the charge accumulation time for camera shake correction imaging may be determined.

  Instead, the time is shorter than the charge accumulation time for camera shake correction imaging determined based on the imaging result (brightness of the object, that is, the output of the integrating unit 11) in the frame before the frame where the shutter button press is detected. It is good also as integrating during this time. In this case, the amount of movement obtained by integration multiplied by the ratio (SS / SG) of the charge accumulation time to the accumulation time SG is obtained as the amount of movement per charge accumulation time (estimated as the amount of movement). )be able to.

  At the same time as the detection Ton of pressing the shutter button, the integration of the output of the camera shake sensor is started, and if the integration is performed over a time shorter than the charge accumulation time, the frame (FP (1)) in which the pressing of the shutter button is detected. ) In the next frame (FP (2)).

Embodiment 2. FIG.
FIG. 23 is a flowchart showing an operation including processing for quickly determining the number of times of reading (the number of divisions in the charge accumulation period) N according to the amount of camera shake. In the first embodiment, as shown in the flowchart of FIG. 20, the movement amount is moved for each divided charge accumulation period (image generation period) by repeating the processing of steps St3 and St4 in FIG. Although the amount is determined, in the second embodiment, the number of times of reading N is calculated from the camera shake amount BL.

FIG. 23 is generally the same as FIG. The only difference is that steps St3, St4 and St5 in FIG. 20 are replaced by a single step St33.
In step St33, a value obtained by dividing the camera shake amount BL by the upper limit value Ln of the movement amount, BL / Ln, which is larger than 2 to the (m−1) th power and satisfies 2 to the mth power is calculated.

The processing from the detection of the shutter button press (St1) to the determination of the number of readings based on the camera shake amount BL (St33) is performed from the rising edge of one VD (first VD) to the next VD (second VD). ), The processing from step St7 to step St13 is the period from the rising edge of the second VD to the rising edge of the next VD (third VD). Run on
The processing from step St15 to the end is executed within a period from the rising edge of the third VD to the rising edge of the next VD (fourth VD).

  The processing from the detection of the shutter button press (St1) to the determination of the number of readings N based on the camera shake amount BL (St33) is completed within the period from the rising edge of the first VD to the rising edge of the second VD. If the process is completed within the period from the rising edge of the second VD to the rising edge of the third VD, the process from step St7 to step St13 is performed from the rising edge of the third VD to the fourth VD. The processes from step St15 to the end are executed in the period up to the rising edge, and are executed in the period from the rising edge of the fourth VD to the rising edge of the next VD (fifth VD).

  As described above, according to the second embodiment, since the number of images to be read out immediately can be determined according to the amount of camera shake, the time until the camera shake correction is performed can be shortened.

Note that, as described above, the photoelectric conversion elements 21 of the photoelectric conversion unit 19 of the solid-state imaging device 2 are covered with different color filters, and the color filters are periodically arranged in a predetermined repeating pattern. The amount of vertical transfer (number of pixels) corresponding to each image generation period is desirably an integer multiple of the number of pixels corresponding to the period of the repetitive pattern of the color filter.
With respect to this point, a case where the periodic arrangement of the color filters is a Bayer type will be described with reference to FIG.
In the case of the Bayer type arrangement, as shown in FIG. 4, green G is arranged in a checkered pattern, and red R and blue B are arranged in a line sequence in the other portions. For example, paying attention to the third column (H = 3) in FIG. 4, the arrangement of the pixels is as R, G, R, G,..., G from the top to the bottom. Both G pixels are arranged every other pixel.

  Therefore, if the amount of movement when moving charges in the vertical transfer unit 22 in each image generation period is an odd number of pixels in the forward or backward direction, for example, G pixels are added to the moved R pixels. Or the R pixel is added to the moved G pixel, which is not appropriate. In order to add signal charges of pixels of the same color, it is necessary to make the amount of charge movement equal to even pixels. In FIG. 19, when moving charges in the vertical transfer unit in the operations TSd, TSg, and TSj, they are moved by 4 pixels, and the same color pixels are added together. For example, when the camera shake amount BL in the first image generation period is 5 pixels, the charge transfer amount needs to be either 4 pixels or 6 pixels. As a specific method, one method is to multiply the maximum natural number not exceeding (BL + 1) / 2 by 2 as the amount of charge transfer. For example, when BL = 5, the amount of charge transfer is 6. Further, when BL = 4.9, the charge transfer amount is 4.

  In addition, when moving charges in the vertical transfer unit 22 in each image generation period, if the charges are moved in the forward direction, that is, in the direction of the horizontal transfer unit 27, the charge is accumulated in the vertical transfer unit 22 in the lowest line. Without being discharged into the horizontal transfer unit 27. On the other hand, when the charge is moved in the reverse direction in the vertical transfer unit 22, there is no horizontal transfer unit 27 above the uppermost line due to the structure of the solid-state imaging device 2, so the charge discharge groove (drain) as shown in FIG. 40 is provided. As a result, even when the charge is moved upward in the vertical transfer unit 22, the charge can be discharged into the charge discharge groove 40 without overflowing the uppermost line.

  Further, instead of providing the charge discharge groove 40 in the upper part of the solid-state imaging device 2 as shown in FIG. 24, the charge is upwardly directed in the vertical transfer unit 22 by partially shielding the photoelectric conversion region on the upper part of the imaging surface. It is also possible to prevent the electric charge from overflowing at the uppermost line when moving to. A schematic diagram in this case is shown in FIG. The shaded area is the light shielding area 42. Although the size of the light-shielding region 42 depends on the amount of charge movement required at the time of camera shake correction, (the amount of charge discharged into the charge discharge groove 40 in FIG. 24) is stored when it is assumed that photoelectric conversion is performed in the light-shielding region. It is necessary to secure an area having a size proportional to the shake amount to be corrected by the camera shake correction so as to be approximately equal to the (charge amount).

  In the above-described embodiment, the lens 1 uses a zoom mechanism that can be driven in the optical axis direction, but a similar effect can be obtained even with a single focus lens. Although the vertical transfer pulse VDP has been described for the case of four phases, the same operation is possible for three phases, and of course the same operation is possible for five phases, six phases, and the like.

It is a block diagram which shows the structure of the imaging device of Embodiment 1 of this invention. It is a figure which shows the structure of the solid-state image sensor used by this invention. It is a figure which expands and shows 4 pixels located in a line in the vertical direction in FIG. It is a figure which shows a Bayer type color filter arrangement | sequence. It is a figure which shows a honeycomb type color filter arrangement | sequence. 3 is a functional block diagram illustrating components for performing camera shake correction in the first embodiment. FIG. 6 is a diagram illustrating an operation of a solid-state imaging device that constitutes a combining unit according to Embodiment 1. FIG. It is a timing chart which shows the vertical transfer pulse VDP applied when transferring an electric charge in the forward direction in a vertical transfer part. It is a figure which shows the electric potential distribution of the electrode of the transfer element of a solid-state image sensor in the case of transferring an electric charge in a forward direction within a vertical transfer part. It is a timing chart which shows the vertical transfer pulse VDP applied when an electric charge is transferred in the reverse direction in a vertical transfer part. It is a figure which shows the electric potential distribution of the electrode of the transfer element of a solid-state image sensor in the case of transferring an electric charge in the reverse direction within a vertical transfer part. (A)-(d) is a figure which shows the example of the captured image containing the direction and angle of camera shake of an imaging device, and camera shake. (A)-(i) is a timing chart which shows operation | movement of the imaging device in Embodiment 1. FIG. (A)-(i) is a timing chart which shows operation | movement of the imaging device in Embodiment 1. FIG. (A)-(i) is a timing chart which shows operation | movement of the imaging device in Embodiment 1. FIG. It is a figure which shows the movement amount with respect to the camera-shake amount which concerns on this invention. FIG. 4 is a diagram illustrating the displacement of the captured image and the subject image on the imaging surface of the solid-state imaging device according to the first embodiment. (A)-(f) is the figure which expanded the period which divided | segmented the electric charge accumulation period in FIG. 13 in the time-axis direction. 6 is a diagram illustrating a charge transfer and charge addition method in a vertical transfer unit according to the first embodiment. FIG. It is a flowchart which concerns on Embodiment 1 of this invention. It is a figure which shows the movement amount with respect to the camera-shake amount which concerns on this invention. (A)-(f) is a timing chart which shows operation | movement of the imaging device in the modification of Embodiment 1. FIG. It is a flowchart which concerns on Embodiment 2 of this invention. It is a figure which shows the solid-state image sensor provided with the electric charge discharge groove. It is a figure which shows the solid-state image sensor provided with the light-shielding area | region.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Lens, 2 Solid-state image sensor, 3 Analog signal processing part, 4 Pixel signal A / D conversion part, 5 Signal processing part, 6 Display means, 7 CPU, 8 Timing generator, 9 Shake sensor, 10 Motion signal A / D conversion Unit, 11 integrating unit, 12 lens driving unit, 21 photoelectric conversion element (PD), 22 vertical transfer unit, 24 readout gate unit, 27 horizontal transfer unit, 28 output amplifier, 31 charge storage unit, 32 motion detection unit, 33 synthesis Means, 36 image shift means, 35 image holding means, 34 reading means, 37 control means, 40 charge discharge groove, 42 light shielding area, PX pixel.

Claims (7)

An imaging device including a photoelectric conversion unit that receives light from a subject and performs photoelectric conversion ,
Reading means for reading an image charge accumulated in the photoelectric conversion unit, N times during the charge accumulation time (N is an integer of 2 or more) is divided into,
Image holding means for temporarily holding the image charges read by the reading means;
Image shifting means for shifting the image charge held in the image holding means;
An image sensor comprising:
Motion detection means for detecting the motion of the image sensor;
The image charge read out for the first time by the reading means and held in the image holding means is shifted by a first movement amount corresponding to the amount of movement detected by the motion detecting means, and the shifted image charge is shifted. to the image charge, the second time to read image charges by the readings out means to perform the process of adding in the image holding unit,
When N is larger than 2, the image charge obtained as a result of addition of the image charges read out n-th time (n is an integer from 2 to (N−1)) by the reading unit, and the n-th shift movement amount of corresponding to the movement amount detected by the movement detector, in the shifted image charges, said first (n + 1) th to the read image charges by the readings out means Control means for repeatedly performing the process of adding by the image holding means from 2 to (N-1) in order,
The image pickup apparatus characterized in that the control means increases the number of times the image charge is read so that the amount of movement of the image charge by the image charge shift means does not exceed a predetermined upper limit value. A period from the start of charge accumulation to the first reading;
The period from each reading to the next reading has substantially the same length,
The amount of movement is represented by the number of pixels of the image sensor corresponding to the amount of movement, and the first movement amount and the nth movement amount according to the amount of movement represent the amount of movement as N. The imaging device according to claim 1, wherein the imaging device is given by dividing by. A CCD imaging device comprising: a vertical transfer unit that transfers image charges accumulated in the photoelectric conversion unit; and a readout gate unit that reads image charges accumulated in the photoelectric conversion unit to the vertical transfer unit;
The reading means is composed of the reading gate unit,
The imaging apparatus according to claim 1, wherein both the image holding unit and the image shift unit are configured by the vertical transfer unit.   The imaging apparatus according to claim 3, wherein the motion transfer unit detects a motion in a direction corresponding to a transfer direction of the vertical transfer unit on the image sensor.   The image pickup apparatus according to claim 1, wherein the number of times image charges are read by the reading unit is a numerical value given as a power of two.   The number of times of readout of image charges by the readout unit is expressed by 2 to the power of M (M is an integer of 2 or more), and the increase in the number of readouts is performed by increasing M by 1. Item 2. The imaging device according to Item 1. The photoelectric conversion unit includes photoelectric conversion elements covered with color filters of different colors, and the color filters are periodically arranged in a predetermined repeating pattern,
The imaging apparatus according to claim 1, wherein the amount of movement in the vertical transfer unit is an integer multiple of the number of pixels corresponding to the period of the repetitive pattern of the color filter.

Images