JP7572926B2 - Image pickup element and image pickup device - Google Patents

Description

The present invention relates to an imaging element and an imaging device.

There is a known digital camera that is capable of high-speed readout by partially reading out pixel signals only from a specific imaging area of an image sensor (see Patent Document 1).

JP 2012-58464 A

In the above conventional technology, pixel signals are read out only partially from a specific imaging area of the image sensor, so when this partial readout is performed, it is not possible to read image signals from the entire imaging area of the image sensor.

An imaging element according to one aspect of the invention includes a first photoelectric conversion unit that converts light into electric charges, a second photoelectric conversion unit that converts light into electric charges and that is disposed adjacent to the first photoelectric conversion unit in a first direction, a third photoelectric conversion unit that converts light into electric charges and that is disposed adjacent to the first photoelectric conversion unit in a second direction intersecting the first direction, a fourth photoelectric conversion unit that converts light into electric charges and that is disposed side by side with the first photoelectric conversion unit in the first direction, and a fourth photoelectric conversion unit that converts light into electric charges and that is disposed side by side with the first photoelectric conversion unit in the first direction. a fifth photoelectric conversion unit arranged adjacent to the fourth photoelectric conversion unit in the second direction; a sixth photoelectric conversion unit that converts light into electric charges and arranged adjacent to the fourth photoelectric conversion unit in the second direction; a seventh photoelectric conversion unit that converts light into electric charges and arranged side by side with the first photoelectric conversion unit in the second direction; an eighth photoelectric conversion unit that converts light into electric charges and arranged adjacent to the seventh photoelectric conversion unit in the first direction; and a photoelectric conversion unit that converts light into electric charges and arranged adjacent to the seventh photoelectric conversion unit in the second direction. a ninth photoelectric conversion unit disposed on the substrate, a first transfer unit for transferring the charge converted by the first photoelectric conversion unit, a second transfer unit for transferring the charge converted by the second photoelectric conversion unit, a third transfer unit for transferring the charge converted by the third photoelectric conversion unit, a fourth transfer unit for transferring the charge converted by the fourth photoelectric conversion unit, a fifth transfer unit for transferring the charge converted by the fifth photoelectric conversion unit, a sixth transfer unit for transferring the charge converted by the sixth photoelectric conversion unit, a seventh transfer unit for transferring the charge converted by the seventh photoelectric conversion unit, and a ninth photoelectric conversion unit disposed on the substrate, a first semiconductor substrate having an eighth transfer unit that transfers the charges converted by the ninth photoelectric conversion unit and a ninth transfer unit that transfers the charges converted by the ninth photoelectric conversion unit; and a semiconductor substrate laminated together with the first semiconductor substrate, the first conversion unit converting a first pixel signal based on the charges converted by the first photoelectric conversion unit, a second pixel signal based on the charges converted by the second photoelectric conversion unit, and a third pixel signal based on the charges converted by the third photoelectric conversion unit into digital signals, a fourth pixel signal based on the charges converted by the fourth photoelectric conversion unit, and a fifth transfer unit that converts the fifth photoelectric conversion unit into digital signals, the fifth photoelectric conversion unit being a first pixel signal based on the charges converted by the first photoelectric conversion unit, a second pixel signal based on the charges converted by the second photoelectric conversion unit, and a third pixel signal based on the charges converted by the third photoelectric conversion unit into digital signals, a second conversion unit that converts into digital signals a fifth pixel signal based on the charges converted by the sixth photoelectric conversion unit and a sixth pixel signal based on the charges converted by the sixth photoelectric conversion unit; a third conversion unit that converts into digital signals a seventh pixel signal based on the charges converted by the seventh photoelectric conversion unit, an eighth pixel signal based on the charges converted by the eighth photoelectric conversion unit, and a ninth pixel signal based on the charges converted by the ninth photoelectric conversion unit; a first accumulation time that accumulates the charges converted by the first photoelectric conversion unit, the second photoelectric conversion unit, and the third photoelectric conversion unit; a second semiconductor substrate having a driver that controls the first transfer unit, the second transfer unit, the third transfer unit, the fourth transfer unit, the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, the eighth transfer unit, and the ninth transfer unit so that at least two of a second accumulation time for accumulating the charges converted in the photoelectric conversion unit, the fifth photoelectric conversion unit, and the sixth photoelectric conversion unit, and a third accumulation time for accumulating the charges converted in the seventh photoelectric conversion unit, the eighth photoelectric conversion unit, and the ninth photoelectric conversion unit are different accumulation times; The first transfer unit is connected to a first transfer control line through which a first transfer control signal for controlling the first transfer unit is output, the second transfer unit is connected to a second transfer control line through which a second transfer control signal for controlling the second transfer unit is output, the third transfer unit is connected to a third transfer control line through which a third transfer control signal for controlling the third transfer unit is output, the fourth transfer unit is connected to a fourth transfer control line through which a fourth transfer control signal for controlling the fourth transfer unit is output, the fifth transfer unit is connected to a fifth transfer control line through which a fifth transfer control signal for controlling the fifth transfer unit is output, the sixth transfer unit is connected to a sixth transfer control line through which a sixth transfer control signal for controlling the sixth transfer unit is output, the seventh transfer unit is connected to a seventh transfer control line through which a seventh transfer control signal for controlling the seventh transfer unit is output, the eighth transfer unit is connected to an eighth transfer control line through which an eighth transfer control signal for controlling the eighth transfer unit is output, and the ninth transfer unit is connected to a ninth transfer control line through which a ninth transfer control signal for controlling the ninth transfer unit is output .

According to the present invention, even if image signals are read from the entire imaging area of the image sensor, pixel signals can be read from a specific imaging area of the image sensor at high speed.

FIG. 2 is a block diagram showing the configuration of a camera. FIG. 2 is a cross-sectional view of an imaging element. 1 is a diagram illustrating a pixel array and unit groups of an imaging chip. FIG. 2 is a circuit diagram corresponding to a unit group of the imaging chip. FIG. 2 is a block diagram showing a functional configuration of an imaging element. 5 is a flowchart illustrating a process in an automatic shooting mode according to the first embodiment. FIG. 2 is a diagram illustrating an example of an imaging surface of an imaging element. FIG. 2 is a diagram illustrating an example of an imaging surface of an imaging element. FIG. 2 is a diagram illustrating an example of an imaging surface of an imaging element. 10 is a flowchart illustrating a process in an automatic shooting mode according to a second embodiment.

--First embodiment--
A first embodiment of the present invention will be described with reference to the drawings.

Fig. 1 is a block diagram showing the configuration of a camera 100, which is an example of an imaging device. The camera 100 has a photographing lens 120 and a camera body 110 as an imaging optical system. The photographing lens 120 is attached to the camera body 110. The camera 100 may have the camera body 110 and the photographing lens 120 integrated into one. The photographing lens 120 guides a subject light beam incident along an optical axis to the image sensor 200. The camera body 110 has the image sensor 200, a control unit 101, a drive unit 102, an image processing unit 103, a work memory unit 104, a recording unit 105, and a display unit 106.

The photographing lens 120 is composed of multiple optical lenses, and focuses the subject light beam from the scene onto the light receiving surface of the image sensor 200. In Figure 1, the photographing lens 120 is represented by a single virtual lens.

The driving unit 102 is a control circuit that performs charge accumulation control such as timing control of the image sensor 200 and control of the imaging area included in the image sensor 200 according to instructions from the control unit 101. The driving unit 102 functions as an image sensor control unit that causes the image sensor 200 to perform charge accumulation and output pixel signals. The driving unit 102 may be partially or entirely mounted on the imaging chip 213 included in the image sensor 200 described below. The driving unit 102 may be partially or entirely mounted on the signal processing chip 211. The control unit 101 may be partially mounted on the imaging chip 213 or the signal processing chip 211.

The image signal output from the imaging element 200 is input to the image processing unit 103. The image processing unit 103 performs various image processes using the work memory unit 104 as a workspace to generate image data. For example, when generating image data in a JPEG file format, compression processing is performed after the image data is generated. The generated image data is recorded in the recording unit 105, and is also converted into a display signal and displayed on the display unit 106 for a preset time.

The image processing unit 103 also has, as functional blocks, a main subject detection unit 111 and a feature region detection unit 112. The functions of these will be described later.

Figure 2 is a cross-sectional view of the image sensor 200 provided in the camera body 110. The image sensor 200 includes an image sensor chip 213 that generates pixel signals corresponding to incident light, a signal processing chip 211 that processes the pixel signals output from the image sensor chip 213, and a memory chip 212 that stores the pixel signals.

The imaging chip 213 is stacked on the signal processing chip 211. The imaging chip 213 is electrically connected to the signal processing chip 211 by bumps 209 having conductivity such as Cu. The signal processing chip 211 is stacked on the memory chip 212. The signal processing chip 211 is electrically connected to the memory chip 212 by bumps 209 having conductivity such as Cu.

As shown in FIG. 2, incident light is mainly indicated by a hollow arrow, and enters the imaging element 200 along the positive Z-axis direction. As shown by the coordinate axes, the left direction on the paper, perpendicular to the Z-axis, is the positive X-axis direction, and the front direction on the paper, perpendicular to the Z-axis and X-axis, is the positive Y-axis direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes in FIG. 2.

One example of the imaging chip 213 is a back-illuminated CMOS image sensor. The imaging chip 213 has, in order from the side where incident light is incident along the positive direction of the Z axis, a microlens 201, a color filter 202, a passivation film 203, a light receiving layer 206, and a wiring layer 208. The light receiving layer 206 has a plurality of photodiodes 204 that accumulate charges according to the incident light, and a readout circuit 205 for reading out pixel signals according to the charges accumulated in the photodiodes 204. The readout circuit 205 includes a plurality of transistors.

The microlenses 201 focus incident light onto the corresponding photodiodes 204. There are multiple types of color filters 202 that transmit different wavelengths, and each has a specific arrangement corresponding to each photodiode 204. The arrangement of the color filters 202 will be described later. The passivation film 203 protects the light receiving layer 206 and also functions as a planarization film.

The wiring layer 208 has wiring 207 for outputting pixel signals read from each photodiode 204 via the readout circuit 205 to the signal processing chip 211. The wiring 207 may be configured as multi-layer wiring, and may also include passive and active elements.

A plurality of bumps 209 are arranged on the surface of the wiring layer 208 opposite the light receiving layer 206. The plurality of bumps 209 are aligned with a plurality of bumps 209 provided on the opposing surface of the signal processing chip 211, and the imaging chip 213 and the signal processing chip 211 are pressed together, etc., so that the aligned bumps 209 are joined together and electrically connected.

Similarly, a plurality of bumps 209 are arranged on the opposing surfaces of the signal processing chip 211 and the memory chip 212. These bumps 209 are aligned with each other, and the signal processing chip 211 and the memory chip 212 are pressed together, etc., so that the aligned bumps 209 are joined together and electrically connected.

The bonding between the imaging chip 213 and the signal processing chip 211 is not limited to Cu bump bonding by solid-phase diffusion, but may be micro-bump bonding by solder melting. The bonding between the signal processing chip 211 and the memory chip 212 is not limited to Cu bump bonding by solid-phase diffusion, but may be micro-bump bonding by solder melting. For example, it is sufficient to provide one bump 209 for each unit group described below. Therefore, the size of the bump 209 may be larger than the pitch between adjacent photodiodes 204. Also, in a peripheral area other than the pixel area where the pixels are arranged, a bump larger than the bump 209 corresponding to the pixel area may be provided.

The signal processing chip 211 has through-silicon vias (TSVs) 210 that connect the circuits provided on the front and back surfaces of the chip to each other. The through-silicon vias 210 are preferably provided in the peripheral region. The through-silicon vias 210 may also be provided in the peripheral region of the imaging chip 213 or in the memory chip 212.

Figure 3 is a diagram for explaining the pixel arrangement of the imaging chip 213 and the unit group 231. In particular, it shows the imaging chip 213 observed from the incident surface side. The imaging chip 213 has a pixel region. A plurality of pixels are arranged in a matrix in the pixel region. In this embodiment, 16 pixels, 4 pixels x 4 pixels adjacent to each other, form one unit group 231. In other words, one unit group 231 is one imaging region, and the imaging chip 213 has multiple imaging regions (unit groups 231). The grid lines in the figure show the concept of forming the unit group 231 by grouping adjacent pixels. The number of pixels forming the unit group 231 is not limited to this, and may be about 2000, for example, 32 pixels x 64 pixels, or may be more or less than that.

As shown in the enlarged partial view of the pixel area, the unit group 231 is arranged in a so-called Bayer array consisting of four pixels: green pixel Gb, green pixel Gr, blue pixel B, and red pixel R. The green pixels Gb and Gr are pixels that have a green filter as the color filter 202 and receive light in the green wavelength band of the incident light. The blue pixel B is a pixel that has a blue filter as the color filter 202 and receives light in the blue wavelength band. The red pixel R is a pixel that has a red filter as the color filter 202 and receives light in the red wavelength band.

The imaging element 200 of this embodiment is configured to perform imaging for a specific unit group 231 under imaging conditions different from those for the other unit groups 231. The imaging conditions include, for example, the frame rate, the pixel thinning rate, the number of pixels added, the pixel signal gain, etc.

The thinning rate of pixels in unit group 231 specifies the number of pixels included in unit group 231 that do not have their pixel signals read out. For example, when the thinning rate is "0", pixel signals are read out from all pixels included in unit group 231, but when the thinning rate is "0.5", pixel signals are read out from half of the pixels included in unit group 231 and not read out from the remaining half. When pixels are thinned out, image information is reduced, but power consumption can be reduced.

The pixel addition number of unit group 231 specifies the number of pixels whose pixel signals are added for the pixels included in unit group 231. For example, when the addition number is "1", pixel signals are read out without addition between adjacent pixels, but when the addition number is "2", pixel signals are added between two adjacent pixels and read out. When pixel signals between pixels are added, the number of pixels can be reduced while maintaining image information, thereby reducing the amount of calculation in subsequent image processing.

Figure 4 is a circuit diagram corresponding to unit group 231 of imaging chip 213. In the figure, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Each transistor described below is included in readout circuit 205 in Figure 2.

As described above, the unit group 231 is formed from 16 pixels. The 16 photodiodes 204 corresponding to each pixel are each connected to a transfer transistor 302, and each gate of each transfer transistor 302 is connected to a TX wiring 307 to which a transfer pulse is supplied. In this embodiment, the TX wiring 307 is commonly connected to each gate of the 16 transfer transistors 302.

The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD formed between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to a Vdd wiring 350 to which a power supply voltage is supplied, and its gate is connected to a reset wiring 306 to which a reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to the gates of the 16 reset transistors 303.

The drain of each amplification transistor 304 is connected to a Vdd wiring 350 to which a power supply voltage is supplied. The source of each amplification transistor 304 is connected to the drain of the corresponding selection transistor 305. The gate of each selection transistor 305 is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. The sources of each selection transistor 305 are connected to a common output wiring 309. The load current source 351 supplies a current to the output wiring 309. That is, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 351 may be provided on the imaging chip 213 side or on the signal processing chip 211 side.

Here, we will explain the flow from the start of charge accumulation to pixel output after accumulation ends. When a reset pulse is applied to the gate of the reset transistor 303 via the reset wiring 306 and at the same time a transfer pulse is applied to the gate of the transfer transistor 302 via the TX wiring 307, the potentials of the photodiode 204 and floating diffusion FD are reset to a reset potential that corresponds to the power supply voltage.

When the application of the transfer pulse is released, the photodiode 204 converts the received incident light into electric charge and accumulates it. Thereafter, when the transfer pulse is again applied to the gate of the transfer transistor 302 with the reset pulse not being applied to the gate of the reset transistor 303, the accumulated electric charge is transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after the charge accumulation. Then, when a selection pulse is applied to the gate of the selection transistor 305 through the decoder wiring 308, a pixel signal corresponding to the change from the reset voltage of the floating diffusion FD to the signal potential is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305.

As shown in the figure, in this embodiment, the reset wiring 306 and the TX wiring 307 are common to the 16 pixels forming the unit group 231. That is, the reset pulse and the transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels forming the unit group 231 start charge accumulation at the same timing and end charge accumulation at the same timing. However, the pixel signal corresponding to the accumulated charge is sequentially applied to each selection transistor 305 by a selection pulse, and is selectively output to the output wiring 309. In addition, the reset wiring 306, the TX wiring 307, and the output wiring 309 are provided separately for each unit group 231. In this embodiment, the TX wiring 307 is commonly connected to each gate of the 16 transfer transistors 302, but is not limited to this, and may be commonly connected to each gate of the transfer transistors 302 included in the pixels arranged in the row direction, or may be connected to each gate of each of the 16 transfer transistors 302. In this embodiment, the reset wiring 306 is commonly connected to the gates of the 16 reset transistors 303, but this is not limited to the above. The reset wiring 306 may be commonly connected to the gates of the reset transistors 303 included in the pixels arranged in the row direction, or may be connected to the gates of each of the 16 reset transistors 303.

In this way, by configuring the circuit based on the unit group 231, the charge accumulation time can be controlled for each unit group 231. In other words, adjacent unit groups 231 can output pixel signals with different charge accumulation times. Furthermore, while one unit group 231 is performing a single charge accumulation, the other unit group 231 can be made to repeat charge accumulation multiple times and output a pixel signal each time, so that these unit groups 231 can output each frame for a video at different frame rates.

Figure 5 is a block diagram showing the functional configuration of the image sensor 200. The multiplexer 411 sequentially selects the 16 photodiodes 204 that form the unit group 231, and outputs the respective pixel signals to the output wiring 309 provided corresponding to the unit group 231. The multiplexer 411 is formed in the image sensor chip 213 together with the photodiodes 204.

The pixel signals output via the multiplexer 411 are subjected to CDS and A/D conversion by a signal processing circuit 412 formed in the signal processing chip 211, which performs correlated double sampling (CDS) and analog-to-digital (A/D) conversion. The A/D converted pixel signals are passed to the demultiplexer 413 and stored in pixel memories 414 corresponding to each pixel. Each pixel memory 414 has a capacity capable of storing pixel signals corresponding to the maximum number of integrations described below. The demultiplexer 413 and pixel memories 414 are formed in the memory chip 212.

The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them on to the downstream image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 211 or in the memory chip 212. The diagram shows the connections for one unit group 231, but in reality, these exist for each unit group 231 and operate in parallel. However, it is not necessary for there to be a arithmetic circuit 415 for each unit group 231; for example, one arithmetic circuit 415 may process sequentially while referring to the values of the pixel memory 414 corresponding to each unit group 231 in order.

As described above, output wiring 309 is provided for each unit group 231. Since the imaging element 200 has an imaging chip 213, a signal processing chip 211, and a memory chip 212 stacked on top of each other, by using bumps 209 for electrical connections between the chips for these output wiring 309, it is possible to route wiring without increasing the size of each chip in the surface direction.

The camera 100 has an automatic shooting mode in which still images are captured automatically. FIG. 6 is a flowchart explaining the flow of processing in this automatic shooting mode. Also, FIGS. 7 to 9 are diagrams showing an example of the imaging surface of the image sensor 200. In FIGS. 7 to 9, grid lines indicate the boundaries of the unit groups 231, and one rectangular area corresponds to the imaging area of one unit group 231. The camera 100 starts the processing shown in FIG. 6 when the automatic shooting mode is instructed by the user via an operating member (not shown).

In step S101, the main subject detection unit 111 of the image processing unit 103 drives the drive unit 102 to obtain image data based on an image signal from the image sensor 200, and detects the main subject 500 included in the image represented by the image data. In this embodiment, the main subject 500 is a person's face. A publicly known method can be used for face detection processing to detect a person's face from an image, so a description thereof will be omitted.

In step S102, the characteristic region detection unit 112 of the image processing unit 103 detects a characteristic region (characteristic region) 510 in the main subject 500 detected by the main subject detection unit 111 based on the image data acquired from the image sensor 200. In this embodiment, the characteristic region 510 is the region of the person's eyes and mouth. A publicly known method can be used for the process of detecting the person's eyes and mouth from the image, so a description thereof will be omitted.

In step S103, the image processing unit 103 sets a collection of unit groups 231 capturing at least a portion of the feature region 510 detected by the feature region detection unit 112 as a feature region unit group group 520. The image processing unit 103 also sets a collection of unit groups 231 not included in the feature region unit group group 520, among the unit groups 231 capturing at least a portion of the main subject 500 detected by the main subject detection unit 111, as a subject unit group group 530. Furthermore, the image processing unit 103 sets a collection of unit groups 231 not included in the feature region unit group group 520 and the subject unit group group 530, among the unit groups 231 in the imaging element 200, as a peripheral unit group group 540.

The control unit 101 sets the frame rate for the feature region as the imaging condition for the unit group 231 included in the feature region unit group group 520. The control unit 101 sets the frame rate for the subject as the imaging condition for the unit group 231 included in the subject unit group group 530. The control unit 101 sets the frame rate for the surroundings as the imaging condition for the unit group 231 included in the surrounding unit group group 540. At this time, the control unit 101 sets the frame rate for the subject frame rate higher than the surrounding frame rate. Also, the control unit 101 sets the frame rate for the feature region frame rate higher than the subject frame rate. That is, the control unit 101 sets the highest frame rate for the feature region unit group group 520 corresponding to the feature region 510. Also, the control unit 101 sets the frame rate for the subject unit group group 530 corresponding to the main subject 500 other than the feature region 510 to a frame rate lower than the frame rate set for the feature region unit group group 520 corresponding to the feature region 510 but higher than the frame rate set for the surrounding unit group group 540 corresponding to the surrounding region. The values of the frame rate for the feature region, the frame rate for the subject, and the frame rate for the periphery may be preset and stored in the camera 100, or the user may be able to change the set values later.

In step S104, the control unit 101 performs AF processing to focus the photographing lens 120 on the main subject 500, and then causes the image sensor 200 to capture an image.

The control unit 101 transmits to the driving unit 102 information identifying the unit groups 231 included in each of the feature region unit group group 520, the subject unit group group 530, and the peripheral unit group group 540, and information on the frame rate set for each. The driving unit 102 drives the image sensor 200 to capture an image in accordance with the information transmitted from the control unit 101. In other words, the driving unit 102 executes charge accumulation and pixel signal output for the pixels included in the unit groups 231 for the feature region unit group group 520, the subject unit group group 530, and the peripheral unit group group 540, in accordance with the frame rate set for each.

For example, suppose that the control unit 101 sets the frame rate for the feature region, the frame rate for the subject, and the frame rate for the periphery to 120 fps, 60 fps, and 30 fps, respectively. In this case, during the time (1/30 sec) for obtaining one frame of image signals from the peripheral unit group group 540, the driving unit 102 obtains two frames of image signals from the subject unit group group 530 and obtains four frames of image signals from the feature region unit group group 520. At this time, the driving unit 102 obtains image signals at different frame rates by separately driving the set of the reset transistor 303, the transfer transistor 302, and the selection transistor 305 of the unit group 231 included in the peripheral unit group group 540, the set of the reset transistor 303, the transfer transistor 302, and the selection transistor 305 of the unit group 231 included in the subject unit group group 530, and the set of the reset transistor 303, the transfer transistor 302, and the selection transistor 305 of the unit group 231 included in the feature region unit group group 520.

In step S105, the image processing unit 103 stores image signals from the feature region unit group group 520 in a predetermined storage area of the work memory unit 104, sequentially for each frame. Similarly, the image processing unit 103 stores image signals from the subject unit group group 530 in a predetermined storage area of the work memory unit 104, sequentially for each frame. Similarly, the image processing unit 103 stores image signals from the peripheral unit group group 540 in a predetermined storage area of the work memory unit 104, sequentially for each frame.

In step S106, the image processing unit 103 reads out image signals of multiple frames of the feature region unit group set 520 from the work memory unit 104, and detects changes in the feature regions 510 by calculating the differences between the multiple frames. The image processing unit 103 then determines the state of the main subject 500 based on the detection results.

8 and 9 are diagrams in which only the region including the main subject 500 is cut out from the imaging plane shown in FIG. 7. FIG. 8 shows a state in which a person is blinking. FIG. 9 shows a state in which a person is changing the direction of their gaze. The image processing unit 103, for example, determines the state of the main subject 500 as the state of the blink as shown in FIG. 8. When determining the state of the blink, the image processing unit 103 determines that a blink has occurred when, for example, the iris of the eye is no longer detected at a location where the iris of the eye was detected. The image processing unit 103 also determines, for example, the direction of the gaze as shown in FIG. 9 as the state of the main subject 500. When determining the direction of the gaze, the image processing unit 103 makes the determination based on, for example, a change in the position of the iris of the eye. The image processing unit 103 may determine the state of the main subject 500 based on the facial expression, for example, determining that the person is smiling if the person's eyes and mouth are smiling.

In step S107, the image processing unit 103 reads out the image signals from the feature region unit group group 520, the subject unit group group 530, and the peripheral unit group group 540 from the work memory unit 104, and generates a moving image (through image) for display based on the image signals. The image processing unit 103 fits a frame based on the image signal from the subject unit group group 530 into a portion of the frame based on the image signal from the peripheral unit group group 540 that corresponds to the subject unit group group 530. The image processing unit 103 also fits a frame based on the image signal from the feature region unit group group 520 into a portion of the frame based on the image signal from the subject unit group group 530 that corresponds to the feature region unit group group 520. In this way, the image processing unit 103 synthesizes the frames of the feature region 510, the main subject 500, and the peripheral region to generate a through image. The display frame rate of the through image may be any of the frame rates for the feature region, the frame rate for the subject, and the frame rate for the peripheral region, or may be another frame rate. In addition, since the number of frames per unit time differs between the feature region 510, the main subject 500, and the peripheral region, the number of frames can be matched by thinning or supplementing frames as appropriate.

In step S108, the image processing unit 103 displays the through image generated in step S107 on the display unit 106. At this time, the image processing unit 103 displays a frame indicating the position of the main subject 500, such as a person's face, and a characteristic area 510, such as the person's eyes or the person's mouth, superimposed on the through image. The image processing unit 103 also displays an icon indicating the state of the main subject 500, such as blinking, gaze direction, and facial expression, determined in step S106, superimposed on the through image. In this way, the position and state of the person's face, eyes, and mouth recognized by the camera 100 can be notified to the user.

In step S109, the image processing unit 103 determines whether or not to record a still image (take a still image) based on the state of the main subject 500 determined in step S106. The image processing unit 103 records a still image when the gaze of the person determined in step S106 is directed toward the camera 100 or when the facial expression of the person determined in step S106 is a smile. At this time, the image processing unit 103 synthesizes frames of the characteristic region, the main subject 500, and the peripheral region to generate a single still image, and records it in the recording unit 105. On the other hand, if the image processing unit 103 determines in step S106 that the person is blinking, for example, it does not record a still image.

In step S110, the control unit 101 determines whether or not to capture the next unit of time. Whether or not to capture the next unit of time is determined based on whether or not the user has instructed to end the automatic capture mode at that point in time. If the control unit 101 determines that the next unit of time is to be captured, it makes a positive judgment in step 110 and returns to step S101 above, whereas if the control unit 101 determines that the next unit of time is not to be captured, it makes a negative judgment in step 110 and ends the processing in FIG. 6.

The "unit time" is a time set in advance in the control unit 101, and is, for example, about several seconds. In the process shown in FIG. 6, when the next unit time starts, the main subject 500 and the characteristic region 510 are detected again in steps S101 and S102, and the characteristic region unit group group 520, the subject unit group group 530, and the peripheral unit group group 540 are updated. This makes it possible to sequentially update the characteristic region unit group group 520, the subject unit group group 530, and the peripheral unit group group 540 in accordance with the movement of the main subject 500 and the characteristic region 510. In addition, when the main subject 500 and the characteristic region 510 are detected again in steps S101 and S102, the captured images of the main subject 500 and the characteristic region 510 captured at a higher frame rate than the peripheral region in the previous unit time can be referenced, so that the main subject 500 and the characteristic region 510 can be detected with high accuracy.

According to the embodiment described above, the following advantageous effects can be obtained.
(1) The camera 100 has a plurality of unit groups 231 (imaging regions), and an imaging chip 213 that generates an image signal according to an optical image incident on the unit groups 231. The camera 100 also has a signal processing chip 211 including a characteristic region detection unit 112 that detects a characteristic region 510 of an image indicated by an image signal output from the imaging chip 213 based on the image signal, and a control unit 101 that controls the unit groups 231, among the plurality of unit groups 231, into which an optical image corresponding to the characteristic region 510 is incident, to perform imaging at a characteristic region frame rate, and controls the unit groups 231 other than the unit groups 231 into which an optical image corresponding to the characteristic region 510 is incident, among the plurality of unit groups 231, to perform imaging at a frame rate different from the characteristic region frame rate. As a result, even if an image signal is read out from the entire imaging region of the imaging element 200, pixel signals can be read out at high speed from a specific imaging region of the imaging element 200 (the unit groups 231 into which an optical image corresponding to the characteristic region 510 is incident), and the recognition accuracy of the characteristic region 510 can be improved. Furthermore, since image signals are acquired at a lower frame rate for areas other than the characteristic area 510, the load on driving the image sensor 200 and image processing can be reduced, and power consumption and heat generation can be suppressed, compared to high-speed readout from all pixels.

(2) The camera 100 determines the state of the main subject 500 based on the image signal output from the unit group 231 corresponding to the feature region 510. This allows the camera 100 to use the result of determining the state of the main subject 500 to capture an appropriate shutter opportunity and automatically record a still image.

--Second embodiment--
Next, a second embodiment of the present invention will be described. The configurations of the camera 100 and the image sensor 200 of the second embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

Figure 10 is a flowchart explaining the flow of processing in the automatic shooting mode in the second embodiment. The camera 100 according to the second embodiment starts the processing shown in Figure 10 when the automatic shooting mode is instructed by the user via an operation member (not shown). Steps S201 to S202 are similar to steps S101 to S102 in the first embodiment. That is, the image processing unit 103 detects a main subject from an image captured by the image sensor 200, and detects a characteristic region from the main subject.

In step S203, the image processing unit 103 sets a feature region unit group group, a subject unit group group, and a peripheral unit group group in the same manner as in the first embodiment. Then, the image processing unit 103 sets the pixel thinning rate or the number of additions of the unit groups 231 included in each of the feature region unit group group, the subject unit group group, and the peripheral unit group group according to the state of the main subject.

As the condition of the main subject, for example, the control unit 101 calculates the focus adjustment state (defocus amount) of the photographing lens 120 with respect to the main subject. A publicly known method can be used to calculate the defocus amount, so a description thereof will be omitted.

When the defocus amount is large (i.e., equal to or greater than a predetermined value) and it can be determined that the main subject is somewhat blurred, it is predicted that fine patterns will not be recognized and the state of the main subject will not be recognized accurately. Therefore, the image processing unit 103 sets the pixel thinning rate or the number of additions to a large value in all unit groups 231. This reduces the number of pixels of the image signal read from the image sensor 200, making it possible to reduce the amount of calculation in the subsequent image processing and suppress the power consumption of the image sensor 200.

On the other hand, when the defocus amount is small (i.e., equal to or less than a predetermined value) and it can be determined that the photographing lens 120 is in focus on the main subject, it is desirable to have a large number of pixels in the image signal read from the image sensor 200 in order to correctly recognize the state of the main subject, but the larger the number of pixels, the larger the amount of calculation and power consumption. Therefore, the image processing unit 103 sets the pixel thinning rate or the number of additions to a smaller value for the feature area unit group group corresponding to the feature area used to recognize the state of the main subject than when the defocus amount is large. As a result, the number of pixels in the image signal from the feature area unit group group is increased, so that the state of the main subject can be recognized with high accuracy. In addition, the image processing unit 103 sets the pixel thinning rate or the number of additions to a larger value for the other subject unit group group and the peripheral unit group group than for the feature area unit group group. As a result, it is possible to prevent the amount of calculation in image processing from increasing overall, and to suppress the power consumption of the image sensor 200. However, when AF operation is performed, the focus is on the face, which is the main subject, so the defocus amount will be small from now on, and the thinning rate and addition number can be set small, just as when the defocus amount is below a certain value. AF operation is performed when the photographer operates the AF activation member or during constant AF operation for video, etc.

In addition, as the condition of the main subject, the image processing unit 103 may calculate the proportion of the imaging surface of the image sensor 200 that the main subject occupies, i.e., the proportion of the size of the main subject relative to the size of the imaging surface.

When the proportion of the imaging surface occupied by the main subject is equal to or greater than a predetermined value (e.g., 30%), the amount of information about the main subject in the captured image is large, so it is predicted that the state of the main subject can be recognized with high accuracy even if the pixel thinning rate or the number of additions is increased to a certain extent. Therefore, the image processing unit 103 sets the pixel thinning rate or the number of additions to a large value in all unit groups 231. This reduces the number of pixels in the image signal read from the image sensor 200, reducing the amount of calculation in subsequent image processing and suppressing power consumption of the image sensor 200.

On the other hand, when the proportion of the imaging surface occupied by the main subject is equal to or less than a predetermined value (for example, 10%), the amount of information about the main subject in the captured image is reduced, making it difficult to recognize fine patterns and therefore difficult to recognize the state of the main subject. Therefore, the image processing unit 103 sets the pixel thinning rate or number of additions to a smaller value in all unit groups 231 than when the proportion of the main subject is large. This increases the number of pixels in the image signal read out from the image sensor 200, making it possible to recognize the state of the main subject with high accuracy.

It is desirable to set an upper limit for the pixel thinning rate or addition number depending on the use of the image. When an image is displayed as a through image, the upper limit is set to a thinning rate or addition number that does not impair the quality of the displayed image. For example, the upper limit is set to a thinning rate or addition number that results in a number of pixels that satisfies the resolution of the display unit 106. When an image is recorded as a still image or a video, the upper limit is set to a thinning rate or addition number that does not impair the quality of the recorded image. For example, the upper limit is set to a thinning rate or addition number that results in a number of pixels that satisfies the recording resolution.

In this way, in step S203, the image processing unit 103 sets the pixel thinning rate or the number of pixels to be added for each of the feature region unit group group, the object unit group group, and the surrounding unit group group according to the condition of the main object.

In step S204, the control unit 101 transmits information identifying the unit groups 231 included in each of the feature region unit group group, the subject unit group group, and the peripheral unit group group, and information on the pixel thinning rate or addition number set for each to the drive unit 102. The drive unit 102 drives the image sensor 200 to capture an image in accordance with the information transmitted from the control unit 101, and outputs an image signal. That is, the drive unit 102 executes charge accumulation of the pixels included in the unit groups 231 and output of image signals in accordance with the pixel thinning rate or addition number set for each of the feature region unit group group 520, the subject unit group group 530, and the peripheral unit group group 540. In this case, the driving unit 102 separately drives the set of the reset transistor 303, the transfer transistor 302, and the selection transistor 305 of the unit group 231 included in the feature region unit group group, the set of the reset transistor 303, the transfer transistor 302, and the selection transistor 305 of the unit group 231 included in the subject unit group group, and the set of the reset transistor 303, the transfer transistor 302, and the selection transistor 305 of the unit group 231 included in the peripheral unit group group, to obtain image signals with different pixel thinning rates or addition numbers.

The processing in steps S205 to S210 is the same as that in steps S105 to S510 in the first embodiment. That is, the image processing unit 103 stores the image signal from the image sensor 200 in the work memory unit 104, and then determines the state of the main subject based on the image signal from the feature region unit group group. The image processing unit 103 records a still image according to the result of this determination.

The image processing unit 103 also generates a through image based on the image signal from the image sensor 200, and displays it on the display unit 106. If the pixel thinning rate or the number of pixels added differs between the feature region unit group group, the subject unit group group, and the peripheral unit group group, the pixel count differs between the feature region, the main subject, and the peripheral region, so the pixel count can be adjusted by thinning or interpolating the pixels as appropriate.

According to the embodiment described above, the following advantageous effects can be obtained.
(1) In the camera 100, the larger the defocus amount of the main subject region, the larger the pixel thinning rate or the number of pixels added when capturing an image of an imaging region (unit group 231) into which an optical image corresponding to the main subject region is incident. In this way, in a situation where the state of the main subject cannot be recognized with high accuracy even if the number of pixels of the captured image is increased, the power consumption and the calculation load can be suppressed by reducing the number of pixels of the captured image.

(2) In the camera 100, the larger the proportion of the main subject region in the image represented by the image signal output from the imaging chip 213, the larger the pixel thinning rate or the number of pixels added when imaging the imaging region (unit group 231) into which the light image corresponding to the main subject region is incident. In this way, in a situation where the state of the main subject can be sufficiently recognized, it is possible to reduce power consumption and computational load by reducing the number of pixels in the captured image.

(Variation 1)
In the above-described embodiment, when a plurality of main subjects (faces) are detected, the image processing unit 103 may select one of the plurality of main subjects as a priority subject (priority subject).

In this case, the image processing unit 103 detects changes in the characteristic areas and determines the state of the main subject (face), similar to step S106 described above. For example, a blinking face is the face of a real person, but a non-blinking face may be a drawing or photograph and not a real person. Therefore, the image processing unit 103 selects the blinking face as the priority subject.

Furthermore, a person who is speaking is considered to have a high priority. Therefore, camera 100 determines whether or not the person is speaking as the state of the main subject. When camera 100 detects a change in the audio signal obtained from the microphone and a change in the shape of the mouth, which is a characteristic region, simultaneously, it determines that the person is speaking. Then, the speaking person (face) is selected as the priority subject.

The control unit 101 performs AF processing to focus the photographing lens 120 on the main subject selected as the priority subject. In addition, for main subjects not selected as priority subjects, detection of characteristic regions is not performed thereafter, and images are captured at the same frame rate as the peripheral regions, for example.

(Variation 2)
In the first embodiment described above, the feature region unit group group corresponding to the feature region is imaged at a higher frame rate than the main subject and the peripheral region, so the charge accumulation time is shorter and the image signal output may be smaller. Therefore, the image processing unit 103 may set a larger gain for the pixel signals read out from the pixels included in the unit group 231 for the feature region unit group group than for the subject unit group group and the peripheral unit group group.

In this case, the driving unit 102 amplifies and outputs pixel signals read from pixels included in the unit group 231 with gains set for the feature area unit group group, the subject unit group group, and the surrounding unit group group, respectively. Since the gain setting value is large for the feature area unit group group, the image signal output is large, and the accuracy of detecting changes in the feature area can be improved.

(Variation 3)
In the first embodiment described above, when setting the characteristic region frame rate for the characteristic region unit groups, the image processing unit 103 may change the set value according to the proportion of the main subject on the imaging surface. The larger the proportion of the main subject on the imaging surface, the lower the set value of the characteristic region frame rate, and the smaller the proportion of the main subject on the imaging surface, the higher the set value of the characteristic region frame rate. When the proportion of the main subject on the imaging surface is large, the amount of information related to the main subject on the captured image is large and changes in the characteristic region are easy to detect, so that the power consumption can be reduced by lowering the characteristic region frame rate. On the other hand, when the proportion of the main subject on the imaging surface is small, the amount of information related to the main subject on the captured image is small and changes in the characteristic region are difficult to detect, so that the recognition accuracy of the state of the main subject can be improved by increasing the feature region frame rate.

(Variation 4)
When multiple human faces are detected as main subjects, if the number of faces is greater than a predetermined value, the amount of information about each face in the captured image is small, and it is predicted that it will be difficult to recognize the state of the face. Therefore, in such a case, the image processing unit 103 may set the pixel thinning rate or the number of additions for the characteristic region to a small value to increase the accuracy of recognizing the state of the face.

In addition, if the position of the face of a person detected as the main subject is near the center of the imaging surface, it is predicted that the captured image is a portrait, there is a large amount of facial information, and the state of the face is easy to recognize. In such a case, the image processing unit 103 may set the pixel thinning rate or the number of additions to a large value overall to reduce power consumption and calculation load.

(Variation 5)
In the first embodiment described above, a frame rate is set for the feature region, the main subject region, and the peripheral region, and in the second embodiment described above, an example is described in which a pixel thinning rate or an addition number is set. However, the imaging conditions to be set do not have to be limited to the examples described in the above embodiments. For example, both a pixel thinning rate and an addition number may be set for the feature region, the main subject region, and the peripheral region. In this case, after the pixels are thinned out, the pixel signals of the remaining pixels are added to further reduce the number of pixels. In addition, both a frame rate and a pixel thinning rate or an addition number may be set.

Furthermore, the frame rate may be the same for the main subject region and the peripheral region, while the frame rate is increased only for the feature region. By increasing the frame rate of the feature region above a predetermined frame rate, the recognition accuracy of the feature region can be improved. Conversely, the frame rate may be the same for the feature region and main subject region, while the frame rate is decreased only for the peripheral region. By maintaining the frame rate of the feature region and main subject region at a predetermined frame rate and decreasing the frame rate of the peripheral region, it is possible to reduce power consumption and the load of image processing while maintaining the recognition accuracy of the feature region and main subject region.

(Variation 6)
In the above embodiment, an example has been described in which the present invention is applied to an automatic shooting mode in which still images are automatically captured, but the present invention may also be applied to a normal shooting mode (i.e., a mode in which a still image is recorded when the user presses the release button). In this case, the camera 100 starts the process of FIG. 6 described above when, for example, the user presses the release button halfway, and performs the processes of steps S101 to S108 as in the above embodiment. Then, when the user presses the release button all the way down, the camera 100 records a still image.

The present invention may also be applied to a video capture mode. In this case, the camera 100 starts the process of FIG. 6 described above when the user presses the record button, for example, and records a video captured by the image sensor 200, and ends the process of FIG. 6 when the user presses the record button again.

(Variation 7)
In the above embodiment, an example in which a human face is detected as the main subject has been described, but other subjects may be detected as the main subject. For example, an animal may be detected as the main subject. In this case, for example, the eyes, mouth, nose, ears, tail, etc. of the animal are used as feature regions and control is performed in the same manner as in the above embodiment. This can improve the recognition accuracy and speed of the state of the animal, etc.

The main subject may also be an artificial object. In this case, characteristic regions are detected from color information, pattern information, and the like of the artificial object. For example, areas with high color or brightness contrast are treated as characteristic regions and control is performed in the same manner as in the above-described embodiment. This makes it possible to improve the recognition accuracy and speed of the state of the artificial object, etc.

(Variation 8)
In the above embodiment, an example has been described in which the image sensor 200 is used that can set image capturing conditions for each unit group 231 composed of a plurality of pixels, but an image sensor that can set image capturing conditions for each pixel individually may be used. In this case, in the above steps S103 and S203, pixel groups corresponding to the feature region, main subject region, and peripheral region, respectively, are set, and a frame rate and a pixel thinning rate or number of additions are set for each pixel group.

(Variation 9)
The imaging device according to the present invention is not limited to a digital camera, but may also include, for example, a mobile phone, a smartphone, a tablet terminal, and the like.

(Variation 10)
In the above-described embodiment, an example has been described in which an image sensor in which a floating diffusion is provided for each pixel is used. However, an image sensor in which a floating diffusion is shared by at least two pixels may be used.

The above description is merely an example, and is in no way limited to the configuration of the above embodiment. In addition, the configuration of each modified example may be appropriately combined with the above embodiment.

100...camera, 101...control unit, 102...drive unit, 103...image processing unit, 111...main subject detection unit, 112...feature region detection unit, 120...photographic lens, 200...imaging element, 204...photodiode, 209...bump, 211...signal processing chip, 213...imaging chip, FD...floating diffusion

Claims (19)

A first photoelectric conversion unit that converts light into an electric charge;
a second photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the first photoelectric conversion unit in the first direction;
a third photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the first photoelectric conversion unit in a second direction intersecting the first direction;
a fourth photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the first direction;
a fifth photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the fourth photoelectric conversion unit in the first direction;
a sixth photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the fourth photoelectric conversion unit in the second direction;
a seventh photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the second direction;
an eighth photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the seventh photoelectric conversion unit in the first direction;
a ninth photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the seventh photoelectric conversion unit in the second direction;
a first transfer unit that transfers the charges converted by the first photoelectric conversion unit;
a second transfer unit that transfers the charges converted by the second photoelectric conversion unit;
a third transfer unit that transfers the charges converted by the third photoelectric conversion unit;
a fourth transfer unit that transfers the charges converted by the fourth photoelectric conversion unit;
a fifth transfer unit that transfers the charges converted by the fifth photoelectric conversion unit;
a sixth transfer unit that transfers the charges converted by the sixth photoelectric conversion unit;
a seventh transfer unit that transfers the charges converted by the seventh photoelectric conversion unit;
an eighth transfer unit that transfers the charges converted by the eighth photoelectric conversion unit;
a first semiconductor substrate having a ninth transfer unit that transfers the charges converted by the ninth photoelectric conversion unit;
A semiconductor substrate laminated together with the first semiconductor substrate,
a first conversion unit that converts a first pixel signal based on the charges converted by the first photoelectric conversion unit, a second pixel signal based on the charges converted by the second photoelectric conversion unit, and a third pixel signal based on the charges converted by the third photoelectric conversion unit into digital signals;
a second conversion unit that converts a fourth pixel signal based on the charges converted by the fourth photoelectric conversion unit, a fifth pixel signal based on the charges converted by the fifth photoelectric conversion unit, and a sixth pixel signal based on the charges converted by the sixth photoelectric conversion unit into digital signals;
a third conversion unit that converts a seventh pixel signal based on the charges converted by the seventh photoelectric conversion unit, an eighth pixel signal based on the charges converted by the eighth photoelectric conversion unit, and a ninth pixel signal based on the charges converted by the ninth photoelectric conversion unit into digital signals;
a second semiconductor substrate having a driver that controls the first transfer unit, the second transfer unit, the third transfer unit, the fourth transfer unit, the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, the eighth transfer unit, and the ninth transfer unit so that at least two of a first accumulation time for accumulating charges converted in the first photoelectric conversion unit, the second photoelectric conversion unit, and the third photoelectric conversion unit, a second accumulation time for accumulating charges converted in the fourth photoelectric conversion unit, the fifth photoelectric conversion unit, and the sixth photoelectric conversion unit, and a third accumulation time for accumulating charges converted in the seventh photoelectric conversion unit, the eighth photoelectric conversion unit, and the ninth photoelectric conversion unit are different accumulation times;
the first transfer unit is connected to a first transfer control line through which a first transfer control signal for controlling the first transfer unit is output;
the second transfer unit is connected to a second transfer control line through which a second transfer control signal for controlling the second transfer unit is output;
the third transfer unit is connected to a third transfer control line through which a third transfer control signal for controlling the third transfer unit is output;
the fourth transfer unit is connected to a fourth transfer control line through which a fourth transfer control signal for controlling the fourth transfer unit is output;
the fifth transfer unit is connected to a fifth transfer control line through which a fifth transfer control signal for controlling the fifth transfer unit is output;
the sixth transfer unit is connected to a sixth transfer control line through which a sixth transfer control signal for controlling the sixth transfer unit is output;
the seventh transfer unit is connected to a seventh transfer control line through which a seventh transfer control signal for controlling the seventh transfer unit is output;
the eighth transfer unit is connected to an eighth transfer control line through which an eighth transfer control signal for controlling the eighth transfer unit is output;
the ninth transfer unit is connected to a ninth transfer control line to which a ninth transfer control signal for controlling the ninth transfer unit is output;
Image sensor. 2. The imaging device according to claim 1,
The drive unit is
at least two of the first accumulation time, the second accumulation time, and the third accumulation time are different from each other,
a timing for outputting the first transfer control signal to the first transfer control line;
a timing for outputting the second transfer control signal to the second transfer control line;
a timing for outputting the third transfer control signal to the third transfer control line;
a timing for outputting the fourth transfer control signal to the fourth transfer control line;
a timing for outputting the fifth transfer control signal to the fifth transfer control line;
a timing for outputting the sixth transfer control signal to the sixth transfer control line;
a timing for outputting the seventh transfer control signal to the seventh transfer control line;
a timing for outputting the eighth transfer control signal to the eighth transfer control line;
and controlling a timing for outputting the ninth transfer control signal to the ninth transfer control line.
Image sensor. 3. The imaging device according to claim 1,
a first floating diffusion to which the charges converted by the first photoelectric conversion unit are transferred;
a second floating diffusion to which the charges converted by the fourth photoelectric conversion unit are transferred;
a third floating diffusion to which the charges converted by the seventh photoelectric conversion unit are transferred;
a first reset unit that resets a potential of the first floating diffusion; a second reset unit that resets a potential of the second floating diffusion; and a third reset unit that resets a potential of the third floating diffusion.
Image sensor. 4. The imaging device according to claim 3 ,
the driving unit controls the first reset unit, the second reset unit, and the third reset unit so that at least two of the first accumulation time, the second accumulation time, and the third accumulation time are different from each other.
Image sensor. 5. The imaging device according to claim 4 ,
The first reset unit is connected to a first reset control line through which a first reset control signal for controlling the first reset unit is output,
the second reset unit is connected to a second reset control line through which a second reset control signal for controlling the second reset unit is output;
the third reset unit is connected to a third reset control line through which a third reset control signal for controlling the third reset unit is output;
The driving unit adjusts the first accumulation time, the second accumulation time, and the third accumulation time so that at least two of the first accumulation time, the second accumulation time, and the third accumulation time are different from each other.
a timing for outputting the first reset control signal to the first reset control line;
a timing for outputting the second reset control signal to the second reset control line;
and controlling a timing for outputting the third reset control signal to the third reset control line.
Image sensor. 3. The imaging device according to claim 1,
a first reset unit that discharges the electric charge converted by the first photoelectric conversion unit;
a second reset unit that discharges the electric charge converted by the second photoelectric conversion unit;
a third reset unit that discharges the electric charge converted by the third photoelectric conversion unit;
a fourth reset unit that discharges the electric charge converted by the fourth photoelectric conversion unit;
a fifth reset unit that discharges the electric charge converted by the fifth photoelectric conversion unit;
a sixth reset unit that discharges the electric charge converted by the sixth photoelectric conversion unit;
a seventh reset unit that discharges the electric charge converted by the seventh photoelectric conversion unit;
an eighth reset unit that discharges the electric charge converted by the eighth photoelectric conversion unit;
a ninth reset unit that discharges the charge converted by the ninth photoelectric conversion unit,
the driving unit controls the first reset unit, the second reset unit, the third reset unit, the fourth reset unit, the fifth reset unit, the sixth reset unit, the seventh reset unit, the eighth reset unit, and the ninth reset unit so that at least two of the first accumulation time, the second accumulation time, and the third accumulation time are different accumulation times from each other.
Image sensor. 7. The imaging device according to claim 6 ,
The first reset unit is connected to a first reset control line through which a first reset control signal for controlling the first reset unit is output,
the second reset unit is connected to a second reset control line through which a second reset control signal for controlling the second reset unit is output;
the third reset unit is connected to a third reset control line through which a third reset control signal for controlling the third reset unit is output;
the fourth reset unit is connected to a fourth reset control line through which a fourth reset control signal for controlling the fourth reset unit is output;
The fifth reset unit is connected to a fifth reset control line through which a fifth reset control signal for controlling the fifth reset unit is output,
The sixth reset unit is connected to a sixth reset control line through which a sixth reset control signal for controlling the sixth reset unit is output,
the seventh reset unit is connected to a seventh reset control line through which a seventh reset control signal for controlling the seventh reset unit is output;
The eighth reset unit is connected to an eighth reset control line through which an eighth reset control signal for controlling the eighth reset unit is output,
The ninth reset unit is connected to a ninth reset control line through which a ninth reset control signal for controlling the ninth reset unit is output,
The drive unit is
at least two of the first accumulation time, the second accumulation time, and the third accumulation time are different from each other,
a timing for outputting the first reset control signal to the first reset control line;
a timing for outputting the second reset control signal to the second reset control line;
a timing for outputting the third reset control signal to the third reset control line;
a timing for outputting the fourth reset control signal to the fourth reset control line;
a timing for outputting the fifth reset control signal to the fifth reset control line;
a timing for outputting the sixth reset control signal to the sixth reset control line;
a timing for outputting the seventh reset control signal to the seventh reset control line;
a timing for outputting the eighth reset control signal to the eighth reset control line;
and controlling a timing for outputting the ninth reset control signal to the ninth reset control line.
Image sensor. The imaging device according to claim 1 ,
A semiconductor substrate laminated together with the first semiconductor substrate,
a first storage unit configured to store the first pixel signal converted into a digital signal by the first conversion unit;
a second storage unit configured to store the second pixel signal converted into a digital signal by the first conversion unit;
a third storage unit configured to store the third pixel signal converted into a digital signal by the first conversion unit;
a fourth storage unit configured to store the fourth pixel signal converted into a digital signal by the second conversion unit;
a fifth storage unit that stores the fifth pixel signal converted into a digital signal by the second conversion unit;
a sixth storage unit that stores the sixth pixel signal converted into a digital signal by the second conversion unit;
a seventh storage unit that stores the seventh pixel signal converted into a digital signal by the third conversion unit;
an eighth storage unit that stores the eighth pixel signal converted into a digital signal by the third conversion unit;
and a ninth storage section that stores the ninth pixel signal converted into a digital signal by the third conversion section. 9. The imaging device according to claim 1,
a first output line to which the first pixel signal, the second pixel signal, and the third pixel signal are output;
a second output line to which the fourth pixel signal, the fifth pixel signal, and the sixth pixel signal are output;
a third output line to which the seventh pixel signal, the eighth pixel signal, and the ninth pixel signal are output,
the first conversion unit converts the first pixel signal, the second pixel signal, and the third pixel signal output to the first output line into digital signals;
the second conversion unit converts the fourth pixel signal, the fifth pixel signal, and the sixth pixel signal output to the second output line into digital signals;
the third conversion unit converts the seventh pixel signal, the eighth pixel signal, and the ninth pixel signal output to the third output line into digital signals.
Image sensor. 10. The imaging device according to claim 9 ,
The second semiconductor substrate is
a first current source that supplies a current to the first output line;
a second current source that supplies a current to the second output line;
and a third current source that supplies a current to the third output line. The imaging device according to any one of claims 1 to 10 ,
the driving unit controls at least two of a first frame rate for reading out the first pixel signal, the second pixel signal, and the third pixel signal, a second frame rate for reading out the fourth pixel signal, the fifth pixel signal, and the sixth pixel signal, and a third frame rate for reading out the seventh pixel signal, the eighth pixel signal, and the ninth pixel signal to be different frame rates from each other.
Image sensor. The imaging device according to any one of claims 1 to 11 ,
the drive unit controls at least two of a first gain for amplifying the first pixel signal, the second pixel signal, and the third pixel signal, a second gain for amplifying the fourth pixel signal, the fifth pixel signal, and the sixth pixel signal, and a third gain for amplifying the seventh pixel signal, the eighth pixel signal, and the ninth pixel signal to be different gains from each other.
Image sensor. The imaging device according to any one of claims 1 to 12 ,
The first semiconductor substrate is
a tenth photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the first direction;
an eleventh photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the tenth photoelectric conversion unit in the first direction;
a twelfth photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the tenth photoelectric conversion unit in the second direction;
a tenth transfer unit that transfers the charges converted by the tenth photoelectric conversion unit;
an eleventh transfer unit that transfers the charges converted by the eleventh photoelectric conversion unit;
a twelfth transfer unit that transfers the charge converted by the twelfth photoelectric conversion unit,
the second semiconductor substrate has a fourth conversion unit configured to convert a tenth pixel signal based on the charges converted in the tenth photoelectric conversion unit, an eleventh pixel signal based on the charges converted in the eleventh photoelectric conversion unit, and a twelfth pixel signal based on the charges converted in the twelfth photoelectric conversion unit into digital signals;
the drive unit controls the first transfer unit, the second transfer unit, the third transfer unit, the fourth transfer unit, the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, the eighth transfer unit, the ninth transfer unit, the tenth transfer unit, the eleventh transfer unit, and the twelfth transfer unit so that at least two of the first accumulation time, the second accumulation time, the third accumulation time, and a fourth accumulation time for accumulating the charges converted in the tenth photoelectric conversion unit, the eleventh photoelectric conversion unit, and the twelfth photoelectric conversion unit are different accumulation times;
the tenth transfer unit is connected to a tenth transfer control line to which a tenth transfer control signal for controlling the tenth transfer unit is output;
the eleventh transfer unit is connected to an eleventh transfer control line through which an eleventh transfer control signal for controlling the eleventh transfer unit is output;
the twelfth transfer unit is connected to a twelfth transfer control line through which a twelfth transfer control signal for controlling the twelfth transfer unit is output;
Image sensor. 14. The imaging device according to claim 13 ,
The first semiconductor substrate is
a thirteenth photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the second direction;
a fourteenth photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the thirteenth photoelectric conversion unit in the first direction;
a fifteenth photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the thirteenth photoelectric conversion unit in the second direction;
a thirteenth transfer unit that transfers the charges converted by the thirteenth photoelectric conversion unit;
a fourteenth transfer unit that transfers the electric charges converted by the fourteenth photoelectric conversion unit;
a 15th transfer unit that transfers the charge converted by the 15th photoelectric conversion unit,
the second semiconductor substrate has a fifth conversion unit configured to convert a thirteenth pixel signal based on the charges converted in the thirteenth photoelectric conversion unit, a fourteenth pixel signal based on the charges converted in the fourteenth photoelectric conversion unit, and a fifteenth pixel signal based on the charges converted in the fifteenth photoelectric conversion unit into digital signals;
the drive unit controls the first transfer unit, the second transfer unit, the third transfer unit, the fourth transfer unit, the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, the eighth transfer unit, the ninth transfer unit, the tenth transfer unit, the eleventh transfer unit, the twelfth transfer unit, the thirteenth transfer unit, the fourteenth transfer unit, and the fifteenth transfer unit so that at least two of the first accumulation time, the second accumulation time, the third accumulation time, the fourth accumulation time, and a fifth accumulation time for accumulating the charges converted in the thirteenth photoelectric conversion unit, the fourteenth photoelectric conversion unit, and the fifteenth photoelectric conversion unit are different accumulation times;
the thirteenth transfer unit is connected to a thirteenth transfer control line through which a thirteenth transfer control signal for controlling the thirteenth transfer unit is output;
the fourteenth transfer unit is connected to a fourteenth transfer control line through which a fourteenth transfer control signal for controlling the fourteenth transfer unit is output;
the fifteenth transfer unit is connected to a fifteenth transfer control line through which a fifteenth transfer control signal for controlling the fifteenth transfer unit is output;
Image sensor. The imaging device according to any one of claims 1 to 12 ,
The first semiconductor substrate is
a tenth photoelectric conversion unit that converts light into electric charges and is arranged alongside the first photoelectric conversion unit in the second direction;
an eleventh photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the tenth photoelectric conversion unit in the first direction;
a twelfth photoelectric conversion unit that converts light into electric charges and is disposed adjacent to the tenth photoelectric conversion unit in the second direction;
a tenth transfer unit that transfers the charges converted by the tenth photoelectric conversion unit;
an eleventh transfer unit that transfers the charges converted by the eleventh photoelectric conversion unit;
a twelfth transfer unit that transfers the charge converted by the twelfth photoelectric conversion unit,
the second semiconductor substrate has a fourth conversion unit configured to convert a tenth pixel signal based on the charges converted in the tenth photoelectric conversion unit, an eleventh pixel signal based on the charges converted in the eleventh photoelectric conversion unit, and a twelfth pixel signal based on the charges converted in the twelfth photoelectric conversion unit into digital signals;
the drive unit controls the first transfer unit, the second transfer unit, the third transfer unit, the fourth transfer unit, the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, the eighth transfer unit, the ninth transfer unit, the tenth transfer unit, the eleventh transfer unit, and the twelfth transfer unit so that at least two of the first accumulation time, the second accumulation time, the third accumulation time, and a fourth accumulation time for accumulating the charges converted in the tenth photoelectric conversion unit, the eleventh photoelectric conversion unit, and the twelfth photoelectric conversion unit are different accumulation times;
the tenth transfer unit is connected to a tenth transfer control line to which a tenth transfer control signal for controlling the tenth transfer unit is output;
the eleventh transfer unit is connected to an eleventh transfer control line through which an eleventh transfer control signal for controlling the eleventh transfer unit is output;
the twelfth transfer unit is connected to a twelfth transfer control line through which a twelfth transfer control signal for controlling the twelfth transfer unit is output;
Image sensor. An imaging device comprising the imaging element according to claim 1 . 17. The imaging device according to claim 16 ,
a detection unit that detects a specific subject from among the subjects imaged by the imaging element,
the drive unit controls the first transfer unit, the second transfer unit, the third transfer unit, the fourth transfer unit, the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, the eighth transfer unit, and the ninth transfer unit based on a detection result of the detection unit so that at least two of the first accumulation time, the second accumulation time, and the third accumulation time are different accumulation times from each other.
Imaging device. 18. The imaging device according to claim 17 ,
The detection unit detects, as the specific subject, a subject that is estimated to be a person from subjects captured by the image sensor. 19. The imaging device according to claim 17 or 18 ,
The detection unit detects, as the specific subject, a subject that is estimated to be a person's eye or a person's mouth from the subjects imaged by the imaging element.

Images