JP7567892B2 - Image sensor and image pickup device - Google Patents

Description

The present invention relates to an imaging element and an imaging device.

An electronic device has been proposed that includes an imaging element in which a back-illuminated imaging chip and a signal processing chip are stacked (hereinafter, referred to as a stacked imaging element) (see Patent Document 1). In the stacked imaging element, the back-illuminated imaging chip and the signal processing chip are stacked so that they are connected via microbumps in each predetermined area.

JP 2006-49361 A

In conventional electronic devices equipped with stacked imaging elements, there have been few proposals to divide an image into blocks having one or more of the above-mentioned regions and capture an image for each block, and electronic devices equipped with stacked imaging elements have not been easy to use. Furthermore, conventional technology that combines multiple images captured separately under different exposure conditions is not suitable for capturing images of moving subjects.

An imaging element according to a first aspect of the invention is an imaging element including a plurality of stacked semiconductor substrates, the plurality of semiconductor substrates including a first photoelectric conversion unit that converts light into electric charges, a second photoelectric conversion unit that converts light into electric charges and that is arranged alongside the first photoelectric conversion unit in the row direction, a third photoelectric conversion unit that converts light into electric charges and that is arranged alongside the first photoelectric conversion unit in the column direction, a first conversion unit that converts a first signal based on the charges converted by the first photoelectric conversion unit into a digital signal, a second conversion unit that converts a second signal based on the charges converted by the second photoelectric conversion unit into a digital signal, a third conversion unit that converts a third signal based on the charges converted by the third photoelectric conversion unit into a digital signal, and a digital signal conversion unit that converts the first signal converted into a digital signal by using the first conversion unit. the first photoelectric conversion unit, the second photoelectric conversion unit, and the third photoelectric conversion unit are disposed on a first semiconductor substrate among the plurality of semiconductor substrates, and the first adder, the second adder, the third adder, the first storage unit, the second storage unit, and the third storage unit are disposed on a semiconductor substrate different from the first semiconductor substrate among the plurality of semiconductor substrates .
An imaging device according to a second aspect of the invention comprises the imaging element according to the first aspect.

According to the present invention, photoelectric conversion data can be appropriately processed.

FIG. 2 is a cross-sectional view of a stacked imaging element. 2 is a diagram illustrating a pixel array and a unit area of an imaging chip. FIG. 3 is a circuit diagram corresponding to a unit area of the imaging chip. FIG. FIG. 2 is a block diagram showing a functional configuration of an imaging element. FIG. 2 is a diagram for explaining the flow of a pixel signal per pixel. FIG. 1 is a block diagram illustrating a configuration of an imaging device having an imaging element. 2 is a diagram illustrating an example of a region of interest and a peripheral region in an imaging element; 3A to 3C are diagrams for explaining readout timing, an accumulation signal, and a pixel signal read out from an image sensor via an arithmetic circuit. 5 is a flowchart illustrating a flow of a photographing operation executed by a control unit according to the first embodiment. 10 is a flowchart illustrating a flow of a photographing operation executed by a control unit according to a second embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
First Embodiment
<Explanation of stacked image sensor>
First, a stacked imaging element 100 mounted on an electronic device (e.g., imaging device 1) according to a first embodiment of the present invention will be described. The stacked imaging element 100 is described in Japanese Patent Application No. 2012-139026, previously filed by the applicant of the present application. FIG. 1 is a cross-sectional view of the stacked imaging element 100. The imaging element 100 includes a back-illuminated imaging chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The imaging chip 113, the signal processing chip 111, and the memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 such as Cu.

As shown in the figure, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface. Also, as shown on the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood, based on the coordinate axes in Figure 1.

One example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back side of the wiring layer 108. The PD layer 106 has a plurality of PDs (photodiodes) 104 arranged two-dimensionally and storing electric charges according to incident light, and transistors 105 provided corresponding to the PDs 104.

A color filter 102 is provided on the incident light side of the PD layer 106 via a passivation film 103. There are multiple types of color filters 102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A combination of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

A microlens 101 is provided on the incident light side of the color filter 102, corresponding to each pixel. The microlens 101 focuses the incident light toward the corresponding PD 104.

The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered and may include passive and active elements.

A number of bumps 109 are arranged on the surface of the wiring layer 108. The bumps 109 are aligned with the bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed together, etc., so that the aligned bumps 109 are joined together and electrically connected.

Similarly, a number of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed together, etc., so that the aligned bumps 109 are bonded together and electrically connected.

The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, but may also employ micro-bump bonding by solder melting. Also, it is sufficient to provide one bump 109 for each unit area, which will be described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD 104. Also, in a peripheral area other than the pixel area where the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel area may also be provided.

The signal processing chip 111 has through silicon vias (TSVs) 110 that connect the circuits provided on the front and back surfaces of the chip. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and in the memory chip 112.

Figure 2 is a diagram illustrating the pixel array of the imaging chip 113 and the unit area 131. In particular, it shows the imaging chip 113 observed from the back side. For example, more than 20 million pixels are arranged in a matrix in the pixel area. In this embodiment, for example, 16 pixels, 4 pixels x 4 adjacent pixels, form one unit area 131. The grid lines in the figure show the concept of forming the unit area 131 by grouping adjacent pixels. The number of pixels forming the unit area 131 is not limited to this, and may be around 1000, for example 32 pixels x 64 pixels, or it may be more or less than that.

As shown in the partially enlarged view of the pixel region, the unit region 131 contains four so-called Bayer arrays, each consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R, arranged vertically and horizontally. The green pixels are pixels that have a green filter as the color filter 102 and receive light in the green wavelength band of the incident light. Similarly, the blue pixels are pixels that have a blue filter as the color filter 102 and receive light in the blue wavelength band, and the red pixels are pixels that have a red filter as the color filter 102 and receive light in the red wavelength band.

In this embodiment, multiple blocks are defined so that each block includes at least one unit area 131, and each block can control the pixels included in the block with different control parameters. In other words, imaging signals with different imaging conditions can be obtained for a pixel group included in one block and a pixel group included in another block. Examples of control parameters include frame rate, gain, thinning rate, number of summation rows or number of summation columns for adding pixel signals, charge accumulation time or number of accumulations, number of digitization bits, etc. Furthermore, the control parameters may be parameters for image processing after image signals are obtained from the pixels.

Figure 3 is a circuit diagram corresponding to unit area 131 of imaging chip 113. In Figure 3, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least some of the transistors described below correspond to transistor 105 in Figure 1.

As described above, the unit region 131 is formed of 16 pixels. The 16 PDs 104 corresponding to each pixel are each connected to a transfer transistor 302, and each gate of each transfer transistor 302 is connected to a TX wiring 307 to which a transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to the 16 transfer transistors 302.

The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and the so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to a Vdd wiring 310 to which a power supply voltage is supplied, and its gate is connected to a reset wiring 306 to which a reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to the 16 reset transistors 303.

The drain of each amplification transistor 304 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. The source of each amplification transistor 304 is connected to the drain of the corresponding selection transistor 305. The gate of each selection transistor 305 is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. The source of each selection transistor 305 is connected to a common output wiring 309. The load current source 311 supplies a current to the output wiring 309. In other words, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

Here, we will explain the flow from the start of charge accumulation to pixel output after accumulation ends. When a reset pulse is applied to the reset transistor 303 via the reset wiring 306 and at the same time a transfer pulse is applied to the transfer transistor 302 via the TX wiring 307, the potentials of the PD 104 and the floating diffusion FD are reset.

When the transfer pulse is released, PD 104 converts the incident light it receives into electric charge and accumulates it. When the transfer pulse is then applied again without the reset pulse being applied, the accumulated electric charge is transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation. Then, when a selection pulse is applied to the selection transistor 305 via the decoder wiring 308, the fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

As shown in FIG. 3, in this embodiment, the reset wiring 306 and the TX wiring 307 are common to the 16 pixels that form the unit area 131. That is, the reset pulse and the transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels that form the unit area 131 start and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output from the output wiring 309 by sequentially applying selection pulses to the respective selection transistors 305. In addition, the reset wiring 306, the TX wiring 307, and the output wiring 309 are provided separately for each unit area 131.

In this way, by configuring the circuit based on the unit area 131, the charge accumulation time can be controlled for each unit area 131. In other words, pixel signals at different frame rates can be output between the unit areas 131. Furthermore, while one unit area 131 is performing a single charge accumulation, the other unit area 131 is caused to repeatedly accumulate charge and output a pixel signal each time, so that each frame for a video can be output at different frame rates between these unit areas 131.

Figure 4 is a block diagram showing the functional configuration of the image sensor 100. An analog multiplexer 411 sequentially selects the 16 PDs 104 that form a unit area 131 and outputs each pixel signal to an output wiring 309 provided corresponding to the unit area 131. The multiplexer 411 is formed in the image sensor chip 113 together with the PDs 104.

The pixel signals output via the multiplexer 411 are subjected to CDS and A/D conversion by a signal processing circuit 412 formed in the signal processing chip 111, which performs correlated double sampling (CDS) and analog-to-digital (A/D) conversion. The A/D converted pixel signals are passed to the demultiplexer 413. The pixel signals output from the demultiplexer 413 are input to the adders 416 corresponding to each pixel. The adder 416 adds the pixel signal output from the demultiplexer 413 and the pixel signal read out from the pixel memory 414 corresponding to each pixel, and outputs the pixel signal after the addition back to the pixel memory 414.

The pixel memories 414 store the pixel signals from the adders 416. Each of the pixel memories 414 has a capacity capable of storing the pixel signals after addition. The demultiplexer 413, the adders 416, and the pixel memories 414 are formed in the memory chip 112.

Figure 5 is a diagram explaining the flow of pixel signals per pixel. In Figure 5, pixel signal S output from demultiplexer 413 is input to the corresponding adder n of adders 416. At this time, pixel signal P stored in the corresponding memory n of pixel memory 414 is read out from memory n and input to adder n.

The adder n adds the input pixel signal S and pixel signal P, and outputs the pixel signal S+P after the addition to the pixel memory n. The pixel memory n stores the input pixel signal S+P and waits for it to be read out to the calculation circuit 415. Here, by controlling the pixel memory 414 so that the pixel signal P stored in the pixel memory n is not read out when the adder n performs the addition, it is possible to output only the pixel signal S input to the adder n directly from the adder n to the pixel memory n. In other words, it is also possible to read out the pixel signal S from the imaging chip 113 directly from the memory n to the calculation circuit 415 without adding it in the adder n.

The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them to a downstream image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112.

The drive control unit 417 generates a timing control signal to synchronize the timing at which pixel signals are sent from the imaging chip 113 to the signal processing chip 111 and the memory chip 112, the timing at which pixel signals are read out and stored in the pixel memory 414, the timing at which pixel signals are added in the adder 416, and the timing at which pixel signals are passed to the arithmetic circuit 415.

Note that while FIG. 4 shows the connections for one unit area 131, in reality these exist for each unit area 131 and operate in parallel. However, a calculation circuit 415 does not need to exist for each unit area 131. For example, one calculation circuit 415 may process sequentially while referring to the values of the pixel memories 414 corresponding to each unit area 131 in order.

As described above, output wiring 309 is provided corresponding to each unit area 131. Since the imaging element 100 has an imaging chip 113, a signal processing chip 111, and a memory chip 112 stacked on top of each other, by using bumps 109 for electrical connections between the chips for these output wiring 309, it is possible to route wiring without increasing the size of each chip in the planar direction.

<Description of Imaging Device>
6 is a block diagram illustrating a configuration of an imaging device 1 having the above-described imaging element 100. In FIG. 6, the imaging device 1 has an imaging optical system 10, an imaging unit 20, an image processing unit 30, a work memory 40, a display unit 50, a recording unit 60, and a control unit 70.

The imaging optical system 10 is composed of multiple lenses, and guides a light beam from the subject field to the imaging unit 20. The imaging optical system 10 may be configured integrally with the imaging device 1, or may be configured to be replaceable with respect to the imaging device 1. The imaging optical system 10 may also include a built-in focus lens or a built-in zoom lens.

The imaging unit 20 has the imaging element 100 described above and a drive unit 21 that drives the imaging element 100. The imaging element 100 is driven and controlled by a control signal output from the drive unit 21, enabling independent accumulation control in block units as described above. The control unit 70 issues instructions to the drive unit 21 regarding the position, shape, range, etc. of the above blocks.

The image processing unit 30 cooperates with the work memory 40 to perform image processing on the image data captured by the imaging unit 20. In this embodiment, the image processing unit 30 performs detection processing of the main subject included in the image in addition to normal image processing (color signal processing, gamma correction, etc.). The detection of the main subject by the image processing unit 30 can be performed using a known face detection function. In addition to face detection, a human body included in the image may be detected as the main subject, as described in, for example, JP 2010-16621 A (US 2010/0002940).

The work memory 40 temporarily stores image data before and after JPEG compression and before and after MPEG compression. The display unit 50 is composed of, for example, a liquid crystal display panel 51, and displays images (still images and videos) captured by the imaging unit 20, various information, and a screen for operation input. The display unit 50 has a configuration in which a touch panel 52 is layered on the display surface of the liquid crystal display panel 51. The touch panel 52 outputs a signal indicating the position where the user touches the liquid crystal display panel 51.

The recording unit 60 stores various data such as image data in a storage medium such as a memory card. The control unit 70 has a CPU and controls the overall operation of the imaging device 1. In this embodiment, the control unit 70 divides the imaging surface of the imaging element 100 (imaging chip 113) into multiple blocks and captures images at different frame rates (accumulation times) and gains between the blocks. To this end, the control unit 70 instructs the driving unit 21 on the position, shape, and range of the blocks, as well as the control parameters for each block.

The control unit 70 also calculates the focus adjustment state of the imaging optical system 10 using the AF calculation unit 71. The control unit 70 further performs exposure calculations using the AE/AWB calculation unit 72 to obtain appropriate exposure.

<Attention Areas and Related Areas>
In this embodiment, the concepts of an attention area and a peripheral area are introduced into the screen to correspond to the above-mentioned multiple blocks. Fig. 7 is a diagram illustrating an attention area 80 and a peripheral area 90 in the image sensor 100 (imaging chip 113). The control unit 70 determines the positions of the attention area 80 and the peripheral area 90 in the image sensor 100 (imaging chip 113) through scene recognition based on a live view image.

Here, the live view image is also called a preview image before actual imaging is performed, and refers to an image for a monitor acquired by the image sensor 100 at a predetermined frame rate (e.g., 30 fps). In FIG. 7, a person is included on the right side of the screen of the live view image, and a tree is included on the left side of the screen of the live view image. The control unit 70 sends an instruction to the image processing unit 30 to perform a known scene recognition process on the live view image data. By performing the scene recognition process, the image processing unit 30 analyzes the live view image and extracts the main subject area.

The image processing unit 30 determines the area including the detected human body as described above as the main subject area. Note that it is also possible to detect animals such as pets and determine the area including the animals as the main subject area. The control unit 70 then determines the main subject area as the attention area 80, and the area outside the attention area 80 as the peripheral area 90.

In addition, when a live view image is displayed on the liquid crystal display panel 51 of the display unit 50, the area of the main subject (displayed) corresponding to the position where the user viewing the live view image touches the touch panel 52 may be set as the attention area 80.

For example, when acquiring a live view image, the control unit 70 causes the AE/AWB calculation unit 72 to perform automatic exposure calculation and determine a white balance adjustment value based on pixel signals output from the attention area 80 and the peripheral area 90 after accumulation for a first accumulation time. The AE/AWB calculation unit 72 calculates exposure (exposure time, gain, etc.) so as to bring the average level of the pixel signals closer to a predetermined level, for example. The AE/AWB calculation unit 72 also determines a white balance adjustment value for expressing white as white.

Furthermore, the control unit 70 generates an image for monitoring based on pixel signals output from the attention area 80 and the peripheral area 90 after accumulation for the first accumulation time, and displays the image on the display unit 50 as the live view image.

The control unit 70 further causes the AF (autofocus) calculation unit 71 to calculate the focus adjustment state of the imaging optical system 10 based on pixel signals output from the attention area 80 after accumulation for the second accumulation time, for example, when acquiring a live view image. In this embodiment, the second accumulation time is controlled to be longer than the first accumulation time.

The AF calculation unit 71 detects the focus adjustment state, for example, by a contrast detection method. Specifically, while moving the position of the focus lens of the imaging optical system 10, the position of the focus lens of the imaging optical system 10 is adjusted so as to increase the contrast of the image formed by the pixel signals output from the attention area 80.

The focus detection process may be performed using a phase difference detection method. In this case, focus detection pixels are provided in advance in the image sensor 100 (image capture chip 113). Then, a phase difference detection calculation is performed using output signals from the focus detection pixels included in the region of interest 80 to detect the focus adjustment state (specifically, the defocus amount) of the image capture optical system 10. The focus detection pixels and phase difference detection calculation are publicly known, as described in, for example, JP 2009-94881 A, and therefore a detailed description will be omitted.

When acquiring a live view image, the control unit 70 sends an instruction to the driving unit 21, and reads out pixel signals having different accumulation times (i.e., pixel signals after the first accumulation time has elapsed and pixel signals after the second accumulation time has elapsed) from the attention area 80 of the image sensor 100 (imaging chip 113) in multiple separate readouts as described above. Here, accumulation control is performed separately for the attention area 80 and the peripheral area 90 before an instruction to perform actual imaging (still image recording or video recording) is given by operating a release switch (not shown).

In other words, until an actual imaging command is issued, exposure calculations, white balance adjustment value determination, and live view image display are performed based on the images acquired in the attention area 80 and the surrounding area 90, and focus detection processing is performed based on the image acquired in the attention area 80.

<Reading out pixel signals with different accumulation times>
Reading of pixel signals having different accumulation times will be described with reference to FIG. 8 which explains the readout timing of pixel signals, accumulation signals in the imaging chip 113, and pixel signals read out from the imaging element 100 via the arithmetic circuit 415.

The driving unit 21 controls the image sensor 100 as follows. That is, in each frame of the live view image, the period from the accumulation start time t0 to time t1 is the first accumulation time, and the period from time t0 to time t2 is the second accumulation time. At time t0, the driving unit 21 starts charge accumulation for the pixels included in the attention area 80 and the peripheral area 90. Then, at time t1, the driving unit 21 outputs pixel signals from the attention area 80 and the peripheral area 90 while controlling the pixel memory 414 so as not to read out the pixel signals stored in the pixel memory n illustrated in FIG. 5. As a result, the pixel signal a accumulated during the first accumulation time (from time t0 to time t1) is output from the demultiplexer 413 and output as it is as signal A via the arithmetic circuit 415. This pixel signal A (=a) is also stored in the pixel memory n.

Furthermore, when the driving unit 21 reads out the pixel signal at time t1, it immediately starts charge accumulation for the pixels included in the attention area 80. Then, at time t2, it controls the pixel memory 414 to read out the pixel signal a stored in the pixel memory n illustrated in FIG. 5, while outputting the pixel signal from the attention area 80. As a result, the pixel signal b accumulated between time t1 and time t2 is output from the demultiplexer 413, and this pixel signal b and the pixel signal a read out from the pixel memory n are added by the adder n. The pixel signal a+b after the addition is output as signal B via the calculation circuit 415. The pixel signal B (=a+b) is the sum of the pixel signals accumulated from time t0 to time t1 and from time t1 to time t2, and therefore corresponds to the pixel signal accumulated during the second accumulation time (time t0 to time t2).

As described above, in the first embodiment, pixel signal B with a high signal level accumulated over the second accumulation time longer than the first accumulation time is used for focus detection processing, so that higher focus detection accuracy can be obtained compared to when pixel signal A with a low signal level is used for focus detection processing. Also, pixel signal A accumulated over the first accumulation time shorter than the second accumulation time is used for exposure calculation, determination of white balance adjustment value, and display of live view image, so that calculations can be performed with high accuracy while avoiding the influence of noise, and a live view image that is not too bright and easy to see can be obtained.

<Explanation of the Flowchart>
Fig. 9 is a flowchart for explaining the flow of the photographing operation executed by the control unit 70 of the imaging device 1 in the first embodiment. When an ON-OFF switch (not shown) is operated to turn on the power and electricity is being supplied to each unit of the imaging device 1, the control unit 70 repeatedly starts the process shown in Fig. 9. In step S101 of Fig. 9, the control unit 70 determines control parameters such as frame rate and gain for the attention area 80 and the peripheral area 90, respectively, and proceeds to step S102. For example, values to be applied in steps S102 and S110 described later are read out from program data and prepared.

In step S102, the control unit 70 sends an instruction to the driving unit 21 to start live view imaging by the imaging unit 20. The acquisition of the live view image that starts in step S102 is performed by setting control parameters for the peripheral region 90 for, for example, substantially the entire imaging surface of the imaging element 100.

In step S103, the control unit 70 sends an instruction to the drive unit 21 to transfer the first data from the imaging unit 20. The first data corresponds to the pixel signal A (=a) read out at the time t1. In step S104, the control unit 70 causes the image processing unit 30 to perform image processing on the live view image data based on the pixel signal A output from the imaging unit 20, and then causes the display unit 50 to display the image.

In step S105, the control unit 70 sends an instruction to the drive unit 21 to transfer the second data from the imaging unit 20. The second data corresponds to the pixel signal B (=a+b) read out at the above-mentioned time t2. In step S106, the control unit 70 causes the AF calculation unit 71 to perform AF calculation (detection of the focus adjustment state) based on the pixel signal B for the AF calculation area. This allows focus adjustment of the imaging optical system 10. Note that since the AF calculation area is not yet determined in the first frame after the power-on operation, the AF calculation is performed for substantially the entire imaging surface of the imaging element 100.

In step S107, the control unit 70 sends an instruction to the image processing unit 30 to extract the main subject area from the live view image. In step S108, the control unit 70 sets the area including the main subject as the attention area 80, determines this attention area 80 as the AF calculation area, and proceeds to step S109. The AF calculation area is applied to the live view image acquired in the next frame.

In step S109, the control unit 70 determines whether or not a release operation has been performed. The release operation is used as a main image capture instruction for the imaging device 1. If the release button (not shown) has been pressed, the control unit 70 makes a positive judgment in step S109 and proceeds to step S110, and if the release button has not been pressed, the control unit 70 makes a negative judgment in step S109 and proceeds to step S112.

When detecting a tap operation on the release icon displayed on the liquid crystal display panel 51 to determine whether or not an instruction to capture an actual image has been issued, step S109 may be judged as positive if the release icon being displayed is tapped.

In step S110, the control unit 70 sends an instruction to the driving unit 21 to set the control parameters (exposure time, gain, etc.) required for the exposure conditions for shooting determined based on the first data (pixel signal A (=a)), and proceeds to step S111.

In step S111, the control unit 70 executes the shooting process, stores the acquired image data in a memory card or the like using the recording unit 60, and returns to step S102. In the shooting process, the control parameters for shooting that are set to be common (the same) across the entire area of the image sensor 100 are applied to acquire and record one still image (main image capture), and also acquire multiple frames of images after acquiring the still image. Then, based on the multiple frames of images acquired within a predetermined time before and after the release operation, slow motion playback video data is generated and recorded. Slow motion playback video data refers to moving image data that is played back at a frame rate (e.g., 15 fps) slower than the frame rate (e.g., 30 fps) when acquired by the image sensor 100.

The control unit 70 generates slow-motion video data as follows. That is, slow-motion video data is generated based on a plurality of frame images (for example, 18 frames for 0.6 seconds when the frame rate is 30 fps) based on the first data (pixel signal A (=a)) temporarily stored in the work memory 40 for displaying the above-mentioned live view image from the time before the release operation time (tx) (for example, 0.6 seconds before) to the time tx, and a plurality of frame images (for example, 12 frames for 0.4 seconds when the frame rate is 30 fps) based on the first data (pixel signal A (=a)) stored in the work memory 40 from the time tx to the time after the time tx (for example, 0.4 seconds after). As a result, slow-motion video data with a playback time of about 2 seconds is generated based on a plurality of frame images (30 images in total) stored in the work memory 40 during the 1 second period (from 0.6 seconds before the time tx to 0.4 seconds after the time tx) that includes the time tx. In this way, slow motion playback video data is obtained based on the frame images acquired before and after the release operation. The slow motion playback video data is generated as MPEG data or JPEG data by the image processing unit 30.

In step S112, which is reached after a negative judgment in step S109 described above, the control unit 70 judges whether or not the power has been turned off. If an ON-OFF switch (not shown) has been operated to turn off the power, the control unit 70 judges step S112 as positive, performs a predetermined power-off process, and ends the process in FIG. 9. If an ON-OFF switch (not shown) has not been operated to turn off the power, the control unit 70 judges step S112 as negative and returns to step S102. If returning to step S102, the above-mentioned process is repeated.

According to the first embodiment described above, the following advantageous effects can be obtained.
(1) The image sensor 100 includes a demultiplexer 413 that inputs data from a plurality of pixels that perform photoelectric conversion, a pixel memory 414 that stores the input data for each pixel, an adder 416 that adds the data stored in the pixel memory 414 and the input data for each pixel, and an arithmetic circuit 415 that outputs the input data or the data added by the adder 416. This allows the photoelectric conversion data to be appropriately processed, improving the ease of use of the image sensor 100.

(2) The pixel memory 414 stores the data added by the adder 416 for each pixel, making it possible to accumulate the input data. This makes it possible to accumulate the photoelectric conversion data and output it, or to output it without accumulating it.

(3) The adder 416 adds data with different photoelectric conversion times for each pixel, so that data equivalent to the case where the photoelectric conversion time is changed to a longer time can be obtained.

(4) The drive control unit 417 is provided to read the data stored in the pixel memory 414 and cause the adder 416 to add the data when data is input from the pixels, allowing control at appropriate timing.

(5) When the data output from the arithmetic circuit 415 is the above-mentioned input data, exposure calculation, determination of a white balance adjustment value, and display of a live view image are performed, and when the data output from the arithmetic circuit 415 is added data, focus detection processing is performed, so that photoelectric conversion data and processing using the same can be appropriately combined.

(6) Whether the arithmetic circuit 415 outputs the input data or the sum data is determined by analyzing the image formed by the input data, so that appropriate photoelectric conversion data can be output according to the image.

(7) In the above determination, additive data is output for pixels corresponding to the main subject region of the image, and input data is output for pixels corresponding to regions other than the main subject region of the image, so that appropriate photoelectric conversion data can be output.

(8) The imaging chip 113 having multiple pixels and the memory chip 112 that processes photoelectric conversion data are stacked, so the imaging element 100 can be configured compactly without increasing the size of each chip in the planar direction.

(9) The imaging device 1, which is an example of an electronic device, includes a memory chip 112 that processes photoelectric conversion data and an imaging chip 113 that has a plurality of pixels, and therefore can provide an imaging device 1 that is easy to use by appropriately processing the photoelectric conversion data.

Second Embodiment
In the first embodiment, an example was described in which the screen is divided into an attention area 80 and a peripheral area 90, and images with different accumulation times are obtained in the attention area 80 and the peripheral area 90 before an instruction for actual imaging is given, but in the second embodiment, the screen is divided into a bright area 85 and a dark area 95, and images with different accumulation times are obtained in the bright area 85 and the dark area 95 after an instruction for actual imaging is given. Here, the bright area 85 is an area formed of pixel signals with values higher than a predetermined pixel signal value (higher than a predetermined luminance value) obtained by analyzing a live view image, for example. The dark area 95 is an area other than the bright area 85.

The imaging device 1 according to the second embodiment acquires an image based on pixel signals accumulated in the bright area 85 for a first accumulation time, and an image based on pixel signals accumulated in the dark area 95 for a second accumulation time, and combines the two images to obtain an image with a wide dynamic range. For this reason, the control unit 70 sends an instruction to the driving unit 21 when acquiring an image for recording, and reads out pixel signals after the first accumulation time has elapsed and pixel signals after the second accumulation time has elapsed from the dark area 95 of the imaging element 100 (imaging chip 113) in multiple separate reads.

<Reading out pixel signals with different accumulation times>
The timing of reading out pixel signals, the accumulated signals in the imaging chip 113, and the pixel signals read out from the imaging element 100 via the arithmetic circuit 415 will be described with reference to FIG.

The driving unit 21 controls the image sensor 100 as follows. That is, the first accumulation time is from the accumulation start time t0 to time t1 of the image for recording after the main imaging instruction, and the second accumulation time is from time t0 to time t2. At time t0, the driving unit 21 starts charge accumulation for the pixels included in the bright area 85 and the dark area 95. Then, at time t1, the driving unit 21 outputs pixel signals from the bright area 85 and the dark area 95 while controlling the pixel memory 414 so as not to read out the pixel signals stored in the pixel memory n illustrated in FIG. 5. As a result, the pixel signal a accumulated during the first accumulation time (from time t0 to time t1) is output from the demultiplexer 413 and output as it is as signal A via the arithmetic circuit 415. This pixel signal A (=a) is also stored in the pixel memory n.

Furthermore, when the driving unit 21 reads out the pixel signal at time t1, it immediately starts charge accumulation for the pixels included in the dark area 95. Then, at time t2, it outputs a pixel signal from the dark area 95 while controlling the pixel memory 414 to read out the pixel signal a stored in the pixel memory n illustrated in FIG. 5. As a result, the pixel signal b accumulated between time t1 and time t2 is output from the demultiplexer 413, and this pixel signal b and the pixel signal a read out from the pixel memory n are added by the adder n. The pixel signal a+b after the addition is output as signal B via the calculation circuit 415. The pixel signal B (=a+b) is the sum of the pixel signals accumulated from time t0 to time t1 and from time t1 to time t2, and therefore corresponds to the pixel signal accumulated during the second accumulation time (time t0 to time t2).

As described above, in the second embodiment, the image of the dark area 95 is formed by the pixel signal B accumulated in the second accumulation time longer than the first accumulation time, so the image is less likely to be crushed black than when the image of the dark area 95 is formed by the pixel signal A with a short accumulation time and a low signal level. Also, since the image of the bright area 85 is formed by the pixel signal A accumulated in the first accumulation time shorter than the second accumulation time, the image is less likely to be blown out white than when the image of the bright area 85 is formed by the pixel signal B with a long accumulation time and a high signal level. As a result, an image with a wide dynamic range from bright areas to dark areas can be obtained by performing the accumulation process from time t0 to time t2 only once. Also, since the accumulation start time is the same for the first accumulation time and the second accumulation time (both are time t0), it is also suitable for shooting a moving subject, unlike the conventional technology in which multiple images with different accumulation start times are synthesized.

<Explanation of the Flowchart>
Fig. 10 is a flowchart for explaining the flow of the photographing operation executed by the control unit 70 of the imaging device 1 in the second embodiment. When an ON-OFF switch (not shown) is operated to turn on the power and electricity is being applied to each unit of the imaging device 1, the control unit 70 repeatedly starts the process shown in Fig. 10. In step S201 of Fig. 10, the control unit 70 determines control parameters such as frame rate and gain for the bright area 85 and the dark area 95, respectively, and proceeds to step S202. For example, values to be applied in steps S202, S205, and S208 described later are read out from program data and prepared.

In step S202, the control unit 70 sends an instruction to the driving unit 21 to start live view imaging by the imaging unit 20. The acquisition of the live view image that starts in step S202 is performed by setting control parameters for the dark area 95 for, for example, substantially the entire imaging surface of the image sensor 100.

In step S203, the control unit 70 determines the bright area 85 and the dark area 95 based on the live view image, and proceeds to step S204. In step S204, the control unit 70 determines whether or not a release operation has been performed. The release operation is used as an instruction to the imaging device 1 to perform actual imaging. If the release button (not shown) has been pressed, the control unit 70 makes a positive judgment in step S204 and proceeds to step S205, and if the release button has not been pressed, the control unit 70 makes a negative judgment in step S204 and proceeds to step S211.

When detecting a tap operation on the release icon displayed on the liquid crystal display panel 51 to determine whether or not an instruction to capture an actual image has been issued, a positive judgment may be made in step S204 if the release icon being displayed is tapped.

In step S205, the control unit 70 sends an instruction to the driving unit 21 to start image capture for recording by the image capture unit 20. The acquisition of the image for recording that starts in step S205 is performed by setting control parameters for the bright area 85 for, for example, substantially the entire imaging surface of the image sensor 100.

In step S206, the control unit 70 sends an instruction to the drive unit 21 to transfer the first data from the imaging unit 20. The first data corresponds to the pixel signal A (=a) read out at the time t1. In step S207, the control unit 70 sends an instruction to the image processing unit 30 to temporarily store the data of the first data corresponding to the bright area 85 in the work memory 40, and the process proceeds to step S208.

In step S208, the control unit 70 sends an instruction to the drive unit 21 to transfer the second data from the imaging unit 20. The second data corresponds to the pixel signal B (=a+b) read out at the time t2. In step S209, the control unit 70 sends an instruction to the image processing unit 30 to temporarily store the second data (i.e., data corresponding to the dark area 95) in the work memory 40, and the process proceeds to step S210.

In step S210, the control unit 70 sends an instruction to the image processing unit 30 to perform image synthesis of the first data and the second data temporarily stored in the work memory 40. Specifically, in the image based on the first data, the area corresponding to the dark area 95 is replaced with the image based on the second data. This results in a synthetic image in which the dark area 95 is composed of pixel signal B (= a + b) and the bright area is composed of pixel signal A (= a). When the image synthesis is completed, the control unit 70 returns to step S202. If returning to step S202, the above-mentioned process is repeated.

In step S211, which is reached after a negative judgment in step S204 described above, the control unit 70 judges whether or not the power has been turned off. If an ON-OFF switch (not shown) has been operated to turn off the power, the control unit 70 judges step S211 to be positive, performs a predetermined power-off process, and ends the process in FIG. 10. If an ON-OFF switch (not shown) has not been operated to turn off the power, the control unit 70 judges step S211 to be negative and returns to step S202.

According to the second embodiment described above, the following advantageous effects can be obtained.
(1) The image sensor 100 forms bright areas 85 of the image when the data output from the arithmetic circuit 415 is data input to the demultiplexer 413, and forms dark areas 95 of the image when the data output from the arithmetic circuit 415 is data added by the adder 416. Therefore, it is possible to obtain an image with a wide dynamic range by appropriately combining photoelectric conversion data and processing using the same.

(2) Whether to output the input data or the sum data from the arithmetic circuit 415 is determined by analyzing the image formed by the input data, so that appropriate photoelectric conversion data can be output according to the image.

(3) In the above determination, the input data is output for pixels corresponding to areas where the image brightness is higher than a predetermined value, and the added data is output for pixels corresponding to areas where the image brightness is lower than a predetermined value, so that appropriate photoelectric conversion data can be output.

(Variation 1)
The imaging device 1 according to the first and second embodiments described above may be configured as a high-performance mobile phone or a tablet terminal. In this case, the camera unit mounted on the high-performance mobile phone (or the tablet terminal) is configured using the stacked imaging element 100 described above.

(Variation 2)
In the above-described second embodiment, an example has been described in which the image processing unit 30 synthesizes images using the first data and the second data temporarily stored in the work memory 40. Instead of this, the image synthesis may be performed using the pixel memory 414 and the adder 416 illustrated in Fig. 4 and Fig. 5. For example, when reading out pixel signals at time t1, control is performed so that only pixel signals A corresponding to the bright area 85 are selectively read out from the image sensor 100. Also, when reading out pixel signals at time t2, control is performed so that only pixel signals B corresponding to the dark area 95 are selectively read out from the image sensor 100.

(Variation 3)
In the first and second embodiments described above, an example has been described in which pixel signals after a first accumulation time has elapsed and pixel signals after a second accumulation time have elapsed are read out in two separate readouts from a predetermined region of the image sensor 100 (image sensor chip 113). The number of readouts is not limited to the above-mentioned two readouts, and may be three, four, or more.

In addition, images read out in multiple instalments before an instruction to capture an actual image may be divided into different applications, such as images to be used for AF calculations, images to be used for exposure calculations and for determining white balance adjustment values, images to be used for extracting the main subject, images to be used for live view display, etc. In addition, images read out in multiple instalments after an instruction to capture an actual image may be divided into different applications, such as extremely bright areas, bright areas, dark areas, extremely dark areas, etc., according to the brightness of the images.

The above description is merely an example, and is in no way limited to the configuration of the above embodiment. The configurations of the above embodiments and modified examples may be combined as appropriate.

REFERENCE SIGNS LIST 1...imaging device 10...imaging optical system 20...imaging section 30...image processing section 40...work memory 50...display section 51...liquid crystal display panel 52...touch panel 60...recording section 70...control section 71...AF calculation section 72...AE, AWB calculation section 100...imaging element 109...bump 111...signal processing chip 112...memory chip 113...imaging chip 131...unit area 413...demultiplexer 414...pixel memory 415...arithmetic circuit 416...adder 417...drive control section

Claims (25)

An imaging element including a plurality of stacked semiconductor substrates,
The plurality of semiconductor substrates include
A first photoelectric conversion unit that converts light into an electric charge;
a second photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the row direction;
a third photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the column direction;
a first conversion unit that converts a first signal based on the charge converted by the first photoelectric conversion unit into a digital signal;
a second conversion unit that converts a second signal based on the charge converted by the second photoelectric conversion unit into a digital signal;
a third conversion unit that converts a third signal based on the charge converted by the third photoelectric conversion unit into a digital signal;
a first adder that performs a first addition process using the first signal that has been converted into a digital signal by the first conversion unit;
a second adder that performs a second addition process using the second signal that has been converted into a digital signal by the second conversion unit;
a third adder that performs a third addition process using the third signal that has been converted into a digital signal by the third conversion unit;
a first storage unit that stores the signal that has been subjected to the first addition process;
a second storage unit that stores the signal that has been subjected to the second addition process;
a third storage unit for storing the signal subjected to the third addition process;
having
the first photoelectric conversion unit , the second photoelectric conversion unit, and the third photoelectric conversion unit are disposed on a first semiconductor substrate among the plurality of semiconductor substrates ;
the first adder, the second adder, the third adder, the first storage unit, the second storage unit, and the third storage unit are disposed on a semiconductor substrate different from the first semiconductor substrate among the plurality of semiconductor substrates .
Image sensor. 2. The imaging device according to claim 1,
a first output line for outputting the first signal to the first conversion unit;
a second output line for outputting the second signal to the second conversion unit;
a third output line for outputting the third signal to the third conversion unit;
An imaging element comprising: 3. The imaging device according to claim 2,
The plurality of semiconductor substrates include
a first load current source that supplies a current to the first output line;
a second load current source that supplies a current to the second output line;
a third load current source that supplies a current to the third output line;
having
the first load current source, the second load current source, and the third load current source are each disposed on the first semiconductor substrate .
Image sensor. 3. The imaging device according to claim 2,
The plurality of semiconductor substrates include
a first load current source that supplies a current to the first output line;
a second load current source that supplies a current to the second output line;
a third load current source that supplies a current to the third output line;
having
an imaging element in which the first load current source, the second load current source, and the third load current source are each disposed on a semiconductor substrate different from the first semiconductor substrate among the plurality of semiconductor substrates; 5. The imaging device according to claim 1,
an accumulation time of the electric charge converted by the first photoelectric conversion unit is controlled to be different from an accumulation time of the electric charge converted by the second photoelectric conversion unit;
Image sensor. 6. The imaging device according to claim 5,
an accumulation time of the electric charge converted by the first photoelectric conversion unit is controlled to be different from an accumulation time of the electric charge converted by the third photoelectric conversion unit;
Image sensor. 5. The imaging device according to claim 1,
an accumulation time of the electric charge converted by the first photoelectric conversion unit is controlled to be different from an accumulation time of the electric charge converted by the third photoelectric conversion unit;
Image sensor. 5. The imaging device according to claim 1,
The plurality of semiconductor substrates include
a first transfer unit that transfers the charges converted by the first photoelectric conversion unit;
a second transfer unit that transfers the charges converted by the second photoelectric conversion unit;
a third transfer unit that transfers the charges converted by the third photoelectric conversion unit;
having
the first transfer unit , the second transfer unit, and the third transfer unit are disposed on the first semiconductor substrate.
Image sensor. 9. The imaging device according to claim 8,
The first transfer unit and the second transfer unit are controlled so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the second photoelectric conversion unit are different accumulation times.
Image sensor. 10. The imaging device according to claim 9,
The first transfer unit and the third transfer unit are controlled so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the third photoelectric conversion unit are different accumulation times.
Image sensor. 9. The imaging device according to claim 8,
The first transfer unit and the third transfer unit are controlled so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the third photoelectric conversion unit are different accumulation times.
Image sensor. 5. The imaging device according to claim 1 ,
The plurality of semiconductor substrates include
a first floating diffusion to which the charges converted by the first photoelectric conversion unit are transferred;
a second floating diffusion to which the charges converted by the second photoelectric conversion unit are transferred;
a third floating diffusion to which the charges converted by the third photoelectric conversion unit are transferred;
A first reset unit that resets the potential of the first floating diffusion;
A second reset unit that resets the potential of the second floating diffusion;
a third reset unit that resets the potential of the third floating diffusion;
having ​
Image sensor. 13. The imaging device according to claim 12,
the first reset unit and the second reset unit perform control so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the second photoelectric conversion unit are different accumulation times.
Image sensor. 14. The imaging device according to claim 13,
the first reset unit and the third reset unit perform control so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the third photoelectric conversion unit are different accumulation times.
Image sensor. 13. The imaging device according to claim 12,
the first reset unit and the third reset unit perform control so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the third photoelectric conversion unit are different accumulation times.
Image sensor. 5. The imaging device according to claim 1 ,
The plurality of semiconductor substrates include
A first reset unit that resets the potential of the first photoelectric conversion unit;
A second reset unit that resets the potential of the second photoelectric conversion unit;
a third reset unit that resets the potential of the third photoelectric conversion unit;
having ​
Image sensor. 17. The imaging device according to claim 16,
the first reset unit and the second reset unit control so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the second photoelectric conversion unit are different accumulation times.
Image sensor. 18. The imaging device according to claim 17,
the first reset unit and the third reset unit perform control so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the third photoelectric conversion unit are different accumulation times.
Image sensor. 17. The imaging device according to claim 16,
the first reset unit and the third reset unit perform control so that an accumulation time of the electric charge converted by the first photoelectric conversion unit and an accumulation time of the electric charge converted by the third photoelectric conversion unit are different accumulation times.
Image sensor. 20. The imaging device according to claim 1,
the first adder, the second adder, the third adder, the first storage unit, the second storage unit, and the third storage unit are disposed on a second semiconductor substrate among the plurality of semiconductor substrates;
Image sensor. 21. The imaging device according to claim 20,
the first conversion unit, the second conversion unit, and the third conversion unit are disposed on a third semiconductor substrate among the plurality of semiconductor substrates;
Image sensor. 22. The imaging device according to claim 21,
the third semiconductor substrate is disposed between the first semiconductor substrate and the second semiconductor substrate in a stacking direction in which the plurality of semiconductor substrates are stacked;
Image sensor. An imaging device comprising the imaging element according to any one of claims 1 to 22 . 24. The imaging device according to claim 23,
An imaging device comprising a control unit that generates image data using at least one of a fourth signal that has been subjected to the first addition processing and stored in the first storage unit, a fifth signal that has been subjected to the second addition processing and stored in the second storage unit, and a sixth signal that has been subjected to the third addition processing and stored in the third storage unit. 24. The imaging device according to claim 23,
an imaging device including a control unit that adjusts the focus of an optical system that directs light to the imaging element, using at least one of a fourth signal that has been subjected to the first addition processing and stored in the first storage unit, a fifth signal that has been subjected to the second addition processing and stored in the second storage unit, and a sixth signal that has been subjected to the third addition processing and stored in the third storage unit.

Images