JP6920983B2 - Image sensor and image sensor - Google Patents

Description

The present invention relates to an image pickup device and an image pickup apparatus.

When an image is taken with a rolling shutter using an image sensor of a CMOS sensor, rolling shutter distortion may occur in the captured image (see, for example, Patent Document 1).
Patent Document 1 Japanese Unexamined Patent Publication No. 2012-199802

When the image is taken with the global shutter, the rolling shutter distortion does not occur, but the influence of the dark current and the read noise in the image sensor may appear in the photographed image.

According to the first aspect of the present invention, a photoelectric conversion unit including a plurality of pixels arranged in the row direction and the column direction, each having a plurality of unit blocks arranged in the row direction and the column direction, and a plurality of unit blocks. Accumulation that executes either batch accumulation control that collectively accumulates pixel signals of all rows of a plurality of pixels or sequential accumulation control that sequentially accumulates pixel signals for each row from a plurality of pixels. Each of the control unit and the plurality of unit blocks is provided with a control selection unit that selects either batch storage control or sequential storage control and causes the storage control unit to execute the control selection unit. The control selection unit has a plurality of storage control units. When batch accumulation control is continued for any of the unit blocks of the above over a predetermined number of times and a predetermined period, the unit blocks are sequentially executed for the accumulation control in the unit block. An image pickup device for selecting the subsequent accumulation control is provided. Further, according to the second aspect of the present invention, a photoelectric conversion unit including a plurality of pixels arranged in the row direction and the column direction, each having a plurality of unit blocks arranged in the row direction and the column direction, and a plurality of photoelectric conversion units. For each unit block, either batch storage control for collectively accumulating pixel signals of all rows of a plurality of pixels or sequential storage control for sequentially accumulating pixel signals for each row from a plurality of pixels is executed. A storage control unit to be used, a control selection unit that selects either batch storage control or sequential storage control for each of a plurality of unit blocks and causes the storage control unit to execute, and a plurality of units arranged in the row direction and the column direction. A special unit block arranged in place of a part of a plurality of unit blocks, including a pixel and a special pixel arranged in place of a part of a plurality of pixels and selectively receiving a part of incident light, and a special unit block arranged in place of a part of a plurality of unit blocks. An image pickup device including a special storage control unit that collectively controls storage for a special unit block and a special information acquisition unit that acquires special information other than brightness related to a subject based on a pixel signal read from a special pixel is provided. NS.

According to the third aspect of the present invention, an image pickup device including the above-mentioned image pickup device is provided.

The outline of the above invention does not list all the necessary features of the present invention. Sub-combinations of these feature groups can also be inventions.

It is a block diagram of the image pickup apparatus 500. It is sectional drawing of the image sensor 100. It is a schematic diagram explaining the layout of a light receiving substrate 113. It is a block diagram of a unit block 131 and a related circuit. It is a circuit diagram of a pixel 150 alone. It is a timing chart which shows the operation of a pixel 150. It is a block diagram of the drive unit 502. It is a schematic diagram which shows the operation of a rolling shutter mode. It is a schematic diagram of the subject 300. It is a schematic diagram which shows the shape of the captured image 311. It is a schematic diagram which shows the operation of a global shutter mode. It is a flow chart which shows the operation of the control selection part 474. It is a schematic diagram of the subject 300. It is a schematic diagram which shows the shape of the captured image 312, 313, 314. It is a schematic diagram which shows the shape of the captured image 322, 323, 324. It is a schematic diagram of the subject 300. It is a schematic diagram which shows the shape of the captured image 315, 316, 317 before correction. It is a flow chart which shows the operation of the image correction part 476. It is a schematic diagram which shows the shape of the corrected image 325, 327.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the inventions that fall within the scope of the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention.

FIG. 1 is a block diagram of the image pickup apparatus 500. The image pickup device 500 includes an optical system including an image pickup lens 520, and an electronic circuit including an image pickup element 100, a system control unit 501, a drive unit 502, a photometric unit 503, a work memory 504, a recording unit 505, and a display unit 506.

The optical system including the image pickup lens 520 guides the incident light flux propagating along the optical axis OA to the image pickup element 100. In the figure, it is represented by one virtual image pickup lens 520 arranged near the pupil, but the optical system includes a plurality of optical lenses and forms an incident light beam on the image pickup surface of the image pickup element 100. .. The image pickup lens 520 may be an interchangeable type that can be attached to and detached from the image pickup apparatus 500.

The drive unit 502 executes pixel signal accumulation control and readout control on the image sensor 100. As a result, the drive unit 502 executes pixel signal accumulation control, read-out control, and the like on the image sensor 100 under the control of the system control unit 501.

The photometric unit 503 has a light receiving element that receives a part of the incident light flux to the image pickup lens 520 and detects the brightness. As a result, the photometric unit 503 detects the brightness distribution of the imaging scene and the like. The light measuring unit 503 may use a part of the image sensor 100 as a light receiving element.

The work memory 504 is used as a work area when the system control unit 501 executes processing. The work memory 504 also functions as a buffer for information to be recorded in the recording unit 505, information displayed by the display unit 506, and the like.

The system control unit 501 includes an image processing unit 511, a calculation unit 512, and the like. The image processing unit 511 executes image processing using the work memory 504 as a work area to generate image data. For example, when the image data to be generated is JPEG format image data, white balance processing, gamma processing, compression processing, and the like are executed.

The image data generated by the image processing unit 511 is recorded in the recording unit 505. Further, the generated image data may be displayed on the display unit 506. Further, it may be transferred to an external storage unit, for example, a cloud server through a communication line. The calculation unit 512 determines, for example, the shutter speed, the aperture value, the ISO sensitivity, and the like in the imaging operation of the imaging device 500 based on the luminance information acquired by the system control unit 501 from the photometric unit 503.

The image pickup device 100 generates a pixel signal corresponding to the incident light flux on the image pickup lens 520. The pixel signal generated by the image sensor 100 becomes image data in the image processing unit 511 of the system control unit 501.

FIG. 2 is a cross-sectional view of the image sensor 100. The image pickup device 100 includes a light receiving substrate 113, a signal processing substrate 111, and a memory substrate 112 that are laminated on each other. The light receiving substrate 113, the signal processing substrate 111, and the memory substrate 112 are electrically connected to each other by a conductive bump 109 such as Cu.

As shown by the white arrows in the figure, the subject luminous flux is incident on the image sensor 100 in the positive direction of the Z axis. In the present embodiment, the surface on which the subject luminous flux is incident on the light receiving substrate 113 will be referred to as the back surface in the following description. Further, as shown by the coordinate axes in the figure, the right direction in the figure orthogonal to the Z axis is described as the X-axis plus direction, and the direction orthogonal to the Z-axis and the X-axis and intersecting the paper surface toward the front is described as the Y-axis plus direction. do.

The light receiving substrate 113 has a photoelectric conversion element layer 106 and a wiring layer 108 as electrical elements. The photoelectric conversion element layer 106 is arranged on the back surface side of the wiring layer 108. The photoelectric conversion element layer 106 has a plurality of photodiodes 104 and a plurality of transistors 105 arranged two-dimensionally. Each of the plurality of transistors 105 is provided corresponding to any one of the plurality of photodiodes 104.

The photoelectric conversion element layer 106 includes a photoelectric conversion element that outputs a pixel signal corresponding to the incident light. An example of the light receiving substrate 113 is a back-illuminated CMOS image sensor, and a photodiode 104 as a photoelectric conversion element is arranged on the photoelectric conversion element layer 106.

In the light receiving substrate 113, the wiring layer 108 has a wiring 107 that transmits a pixel signal from the photoelectric conversion element layer 106 to the signal processing substrate 111. The wiring 107 may be arranged in multiple layers, and may further include a passive element and an active element.

A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with the plurality of bumps 109 provided on the facing surfaces of the signal processing board 111, and the light receiving board 113 and the signal processing board 111 are aligned by pressurizing or the like. The bumps 109 are joined together and electrically connected.

Further, the light receiving substrate 113 has a color filter 102 and a microlens 101 as optical elements. In the light receiving substrate 113, the color filter 102 is arranged on the incident side of the incident light in the photoelectric conversion element layer 106 via the passivation film 103. Therefore, incident light is incident on each of the plurality of photodiodes 104 through the color filter 102.

Further, the color filter 102 includes a plurality of types that transmit different wavelength regions. The different types of color filters 102 have a specific arrangement corresponding to each of the photodiodes 104. The pair of the color filter 102, the photodiode 104, and the transistor 105 forms one pixel.

The microlens 101 is arranged on the incident side of the incident light with respect to the color filter 102 corresponding to each pixel. Each of the microlenses 101 collects incident light towards the corresponding photodiode 104.

The signal processing board 111 processes the pixel signal acquired from the light receiving board 113. In addition, the memory board 112 stores pixel signals. A plurality of bumps 109 are arranged on the surfaces of the signal processing board 111 and the memory board 112 facing each other. These bumps 109 are aligned with each other, and the signal processing board 111 and the memory board 112 are pressurized, so that the aligned bumps 109 are joined to each other and electrically connected.

The bonding between the bumps 109 is not limited to Cu bump bonding by solid phase diffusion, and micro bump bonding by solder melting may be adopted. Further, for example, about one bump 109 may be provided for one output wiring described later. Therefore, the size of the bumps 109 may be larger than the pitch of the photodiode 104. Further, in the peripheral region other than the pixel region in which the pixels are arranged, a bump larger than the bump 109 corresponding to the pixel region may be provided together.

The signal processing board 111 has TSVs (Through Silicon Vias) 110 that connect circuits provided on the front and back surfaces to each other. The TSV 110 is preferably provided in the peripheral region. Further, the TSV 110 may also be provided in the peripheral region of the light receiving substrate 113 and the memory substrate 112.

FIG. 3 is a schematic view showing the layout of the pixels 150 and the unit block 131 in the photoelectric conversion element layer 106 of the image sensor 100. The layout in the figure shows a state in which the light receiving substrate 113 is viewed from the back surface side.

A plurality of unit blocks 131 arranged in the row direction and the column direction are arranged in the photoelectric conversion element layer 106. Further, each of the unit blocks 131 includes a plurality of pixels 150 arranged in the same row direction and column direction as the unit block 131. Therefore, 20 million or more pixels 150 are arranged in a matrix on the image sensor 100.

In the illustrated example, each of the unit blocks 131 is formed by 16 pixels 150 of 4 pixels × 4 pixels adjacent to each other. As described in the partially enlarged view shown in the drawing, the 16 pixels 150 forming the unit block 131 are four pixels of green pixels Gb, Gr, blue pixels B and red pixels R according to, for example, a Bayer arrangement. Either can be assigned.

The green pixels Gb and Gr have a green filter as the color filter 102, and receive light in the green wavelength band among the incident light. Similarly, the blue pixel B has a blue filter as a color filter 102 and receives light in the blue wavelength band, and the red pixel R has a red filter as the color filter 102 and receives light in the red wavelength band. ..

Needless to say, the number of pixels 150 forming the unit block 131 is not limited to the above. Further, the shape of the unit block 131 formed by the plurality of pixels 150 is not limited to a square. Therefore, when the total number of pixels of the light receiving substrate 113 is about 20 million pixels, the layout may be such that the number of arrangements of the unit blocks 131 is, for example, 64 rows and 32 columns, 48 rows and 114 columns, and the like.

FIG. 4 shows an outline of the connection relationship with one of the unit blocks 131 formed in the image sensor 100. The illustrated unit block 131 includes pixels 150 in rows L and columns P, and a row control unit 200 and peripheral circuits 133 are connected to the unit block 131.

In the image pickup device 100, the row control unit 200 is coupled to the unit block 131 through the wiring Rst_l (where l is an integer from 1 to L), the wiring Tx_l, and the wiring Sel_l. Each of the wiring Rst_l, the wiring Tx_l, and the wiring Self_l is commonly connected to the P pixels 150 in the first row in the unit block 131. Similarly, the wiring Tx_l and the wiring Self_l are also commonly connected to the P pixels 150 in the first row in the unit block 131.

The row control unit 200 may be called a row selection unit, a vertical scanning circuit, or the like. The row control unit 200 is provided for each unit block 131. In the image pickup device 100, the row control unit 200 may be provided on the signal processing board 111 side.

In the image sensor 100, the peripheral circuit 133 is coupled to the unit block 131 through the column transmission line 170_p (where p is an integer from 1 to P). The column transmission line 170_p is provided for each pixel 150 in the same row.

Further, the column transmission line 170_p is commonly connected to the L pixels 150 in the p-th row in the unit block 131. As a result, the column transmission line 170 is shared by the pixels 150 in the same row in the unit block 131, and the pixel signals from the pixels 150 included in the row are transmitted.

In the image sensor 100, the column transmission line 170_p is connected to the peripheral circuit 133 mounted on the signal processing board 111 from the light receiving board 113 side through the bump 109. The peripheral circuit 133 on the signal processing board 111 is arranged for each unit block 131 at a position corresponding to the unit block 131 on the light receiving board 113.

Each of the peripheral circuits 133 has an analog / digital conversion circuit 202 and a CDS circuit 204 connected in series for each column transmission line 170_p. The pair of the analog / digital conversion circuit 202 and the CDS circuit 204 is provided with P pieces per unit block 131, which is the same as the number of columns.

The peripheral circuit 133 further has a shift register 206 arranged on the output side of the CDS circuit 204. In the illustrated example, the shift register 206 is arranged for each unit block 131. The output of the shift register 206 is connected to the pixel memory 414 through the column bus line 172. The shift register 206 may be called a horizontal scanning circuit, a multiplexer, or the like.

FIG. 5 is a circuit diagram showing the structure of the pixels 150 forming the unit block 131. However, the circuit shown includes two photodiodes 104-1 and 104-2, and corresponds to two pixels 150. These two pixels are included in the same column in the unit block 131.

In the photodiodes 104-1 and 104-2, the cathode is individually coupled to one end of the selection transistors 61-1 and 61-2 arranged corresponding to each of the photodiodes 104-1 and 104-2. Photodiodes 104-1 and 104-2 each accumulate electric charges according to the amount of light received.

The selection transistors 61-1 and 61-2 switch whether or not to output a voltage according to the amount of electric charge accumulated by the photodiodes 104-1 and 104-2. The conduction state of the selection transistor 61 is controlled by, for example, a pixel selection signal given through the wiring Tx_l given by the row control unit 200.

Further, the pixel 150 includes a reset transistor 59, an amplification transistor 60, a ground side transistor 65, a first transistor 78, a second transistor 63, an amplification transistor 66, and a transfer transistor 64, which are commonly provided in the photodiodes 104-1 and 104-2. , And a capacitor 10. These commonly provided transistors function as a switch unit that controls the state of the capacitor 10 (a state indicating what kind of voltage is applied to both poles).

In the illustrated example, the pixel 150 sequentially holds the feedthrough voltage and the signal voltage at the respective outputs of the photodiodes 104-1 and 104-2 by one capacitor 10. As a result, the area of the CDS circuit of each pixel 150 can be reduced while providing the CDS circuit for each of the pixels 150. Further, since the feedthrough voltage and the signal voltage are held by the common capacitor 10, the pixel value can be detected with high accuracy.

The reset transistor 59 is connected to the photodiode 104 via the respective selection transistor 61. The reset transistor 59 switches whether or not the cathodes of the photodiodes 104-1 and 104-2 are connected to the reference voltage VCS. The reset transistor 59 is controlled by a reset signal applied through the wiring Rst_l of the row control unit 200.

The gate of the amplification transistor 60 is connected to the photodiodes 104-1 and 104-2 via the selection transistors 61-1 and 61-2, respectively. The amplification transistor 60 amplifies the voltage output by either the selected photodiodes 104-1 or 104-2 when the selection transistor 61 is turned on. The amplification transistor 60 is arranged between the reference voltage VCS and the ground potential.

Further, a ground side transistor 65 is arranged between the amplification transistor 60 and the ground potential. A P control voltage is applied to the gate of the ground side transistor 65.

The capacitor 10 has three electrodes, the first electrode and the second electrode are arranged so as to face each other, and the second electrode and the third electrode are arranged so as to face each other. As a result, the capacitor 10 forms a three-terminal capacitor, and is charged and discharged with a voltage corresponding to the amount of electric charge stored in the photodiodes 104-1 and 104-2.

The first electrode of the capacitor 10 is connected to the output end of the amplification transistor 60 via the first transistor 78. The first transistor 78 functions as a first switch for switching whether or not to connect the outputs of the photodiodes 104-1 and 104-2 amplified by the amplification transistor 60 to the first electrode. The first transistor 78 is controlled by the first control signal SH1 applied to the gate.

The second electrode of the capacitor 10 is connected to the output end of the amplification transistor 60 via the second transistor 63. The second transistor 63 functions as a second switch for switching whether or not the output of the photodiode 104 amplified by the amplification transistor 60 is connected to the second electrode. The second transistor 63 is controlled by the second control signal SH2 applied to the gate.

The third electrode of the capacitor 10 is connected to the reference potential. The capacitance between the second electrode and the third electrode maintains the potential of the second electrode with respect to the reference potential. The capacitor 10 has a first electrode, a second electrode, and a third electrode formed so as to be overlapped in the same region. Therefore, the mounting area can be reduced as compared with the case where a plurality of two-terminal capacitors are used. However, the capacitor 10 may be formed by using two elements of a two-terminal capacitor provided between the node B and the node C and a two-terminal capacitor provided between the node B and the reference potential.

The amplification transistor 66 amplifies and outputs the voltage at the first electrode of the capacitor 10. The transfer transistor 64 functions as a transfer unit for switching whether or not to transfer the voltage output by the amplification transistor 66 to the read line 170-p. The transfer transistor 64 is controlled by a row selection signal given from the row control unit 200 through the wiring Self_l.

The capacitor 10 sequentially transitions between a first state in which a feedthrough voltage is applied to the second electrode and a second state in which a signal voltage is applied to the second electrode according to each control signal given from the row control unit 200. Let me. The peripheral circuit 133 turns on the transfer transistor 64 in at least the second state to transfer the voltage level of the first electrode of the capacitor 10. As a result, the peripheral circuit 133 can calculate the pixel value of the pixel signal based on the transferred voltage level.

FIG. 6 is a timing chart showing the operation of the pixel 150, and shows the signal waveforms at the nodes A, B, and C shown in FIG. Node A corresponds to the output end of the amplification transistor 60, node B corresponds to the second electrode of the capacitor 10, and node C corresponds to the first electrode of the capacitor 10.

A case where the output of the photodiode 104-1 is read out will be described. First, the reset signal RST becomes H level, and the reset transistor 59 is turned on. Further, the selection transistor 61-1 is controlled to be in the ON state. As a result, the cathode voltage of the photodiode 104-1 is reset, and the voltage at the node A becomes the reference voltage VCS. The amplification factor of the amplification transistors 60 and 66 is 1.

When time has passed since the reset, the voltage at the node A becomes the feedthrough voltage Vft. The row control unit 200 sets the first control signal SH1 and the second control signal SH2 to the H level after a predetermined time has elapsed after inputting the reset signal RST. As a result, both the first transistor 78 and the second transistor 63 are turned on. At this time, the reference voltage VCS is applied to the first electrode (node C) of the capacitor 10, and the feedthrough voltage Vft is applied to the second electrode (node B) (first state).

Next, the first control signal SH1 and the second control signal SH2 are set to the L level. As a result, the first electrode of the capacitor 10 is in a floating state, and the voltage between the electrodes of the capacitor 10 is maintained.

Subsequently, at the timing when the light receiving of the frame in the photodiode 104-1 ends, the voltage at the node A becomes a voltage corresponding to the voltage sig1 according to the amount of accumulated charge in the photodiode 104-1. Specifically, the voltage is lower than the feedthrough voltage Vft by the voltage sig1.

Further, the second control signal SH2 is set to the H level while maintaining the first control signal SH1 at the L level. As a result, the second transistor 63 is turned on (second state).

Since the first control signal SH1 is maintained at the L level, when the capacitor 10 transitions to the second state, the first electrode is in the floating state. As a result, the signal voltage output by the photodiode 104-1 is applied to the second electrode while maintaining the voltage between the electrodes of the capacitor 10 in the first state. Since the voltage of the second electrode is lowered by the voltage sig1 as compared with the first state, the voltage of the first electrode is also lowered by the voltage sig1 as compared with the first state.

Peripheral circuit 133 reads the voltage of the first electrode (node C) at least in the second state. Since the voltage of the first electrode in the first state is the reference voltage VCS, the peripheral circuit 133 does not have to read the voltage of the first electrode in the first state when the reference voltage is known. In this case, the peripheral circuit 133 detects the voltage sig1 by subtracting the voltage of the first electrode in the second state from the known reference voltage VCS.

The peripheral circuit 133 may correct the value of the known reference voltage VCS according to the temperature of the image sensor 100 or the like. The peripheral circuit 133 may read the voltage of the first electrode (node C) and calculate the difference in each of the first state and the second state. As a result, the voltage sig1 corresponding to the amount of accumulated charge in the photodiode 104-1, that is, the pixel value is calculated.

Next, the output of the photodiode 104-2 is read out. At this time, the reset signal RST becomes H level, and the reset transistor 59 is turned on. Further, the selection transistor 61-2 is controlled to be in the ON state. As a result, the cathode voltage of the photodiode 104-2 is reset, and the voltage at the node A becomes the reference voltage VCS. Subsequent operations are the same as for the photodiode 104-1. The peripheral circuit 133 calculates the voltage sig2 according to the amount of accumulated charge in the photodiode 104-2 from the voltage read out in the second state.

With such a configuration and operation, the outputs of the photodiodes 104-1 and 104-2 can be double-sampled by correlation using one capacitor 10. Therefore, the circuit area can be reduced, and the error due to the variation of the capacitor can be eliminated. Further, in this embodiment, since one capacitor 10 and a peripheral switch portion are shared by the two photodiodes 104, the circuit area can be further reduced.

The capacitor 10 and the switch unit are provided corresponding to the respective photodiode 104, but as shown in the figure, a set of the capacitor 10 and the switch unit are provided for the plurality of photodiodes 104-1 and 104-2. It may be provided in common. Further, a set of capacitors 10 and a switch unit may be provided for one photodiode 104.

A switch unit is provided for each capacitor 10. When the capacitor 10 is shared by two or more photodiodes 104, the capacitor 10 holds the output values of the photodiodes 104-1 and 104-2, respectively, in order. Further, the plurality of capacitors 10 may simultaneously hold the output values of the corresponding photodiodes 104-1 and 104-2.

FIG. 7 is a block diagram of a drive unit 502 that drives the image sensor 100. The drive unit 502 includes a drive control unit 420, a sensor control unit 441, a block control unit 442, a synchronization control unit 443, a signal control unit 444, and a pixel memory 414.

In the drive unit 502, the sensor control unit 441 transmits a control signal related to charge accumulation and charge readout of each pixel in the pixel 150 of the image sensor 100 to the light receiving substrate 113. More specifically, the sensor control unit 441 transmits a reset signal and a pixel selection signal to the row control unit 200 corresponding to each of the unit blocks 131, and starts and ends the charge accumulation of the pixel 150 to be controlled. And control. Further, by transmitting a column selection signal or the like to the read pixels, the pixel signal is output to the column transmission line 170.

In the drive unit 502, the block control unit 442 transmits a control signal for identifying the unit block 131 to the light receiving board 113. More specifically, the block control unit 442 transmits a specific signal that identifies the unit block 131 to be controlled.

In the unit block specific signal, the transfer pulse or the like received by each of the pixels 150 via the wiring Tx_j or the like is a logical product of the selection signal transmitted by the sensor control unit 441 and the specific signal transmitted by the block control unit 442. As a result, the pixels 150 arranged substantially evenly on the light receiving substrate 113 can be controlled as blocks that are independent of each other for each region.

The block control unit 442 can simultaneously transmit measurement signals to the plurality of unit blocks 131. As a result, the image sensor 100 can operate the plurality of unit blocks 131 in synchronization with each other, or can execute the operation across the plurality of unit blocks 131.

In the drive unit 502, the synchronization control unit 443 supplies the synchronization signal to the light receiving substrate 113. As a result, a plurality of elements mounted on the light receiving substrate 113 can be activated in synchronization. Therefore, for example, by adjusting the synchronization signal, it is possible to execute random control, thinning control, and the like in which only specific pixels of pixels belonging to the same unit block 131 are controlled.

In the drive unit 502, the signal control unit 444 is responsible for timing control for the analog / digital conversion circuit 202, the CDS circuit 204, and the shift register 206. As a result, the image processing unit 511 can generate one image data at high speed and with high quality from the pixel signals read from the plurality of unit blocks 131.

In the drive unit 502, the pixel memory 414 has a memory space for storing a pixel signal read from the pixel 150 and digitized. The pixel memory 414 may be provided corresponding to each unit block 131. The I / F circuit 418 is responsible for communication between the drive control unit 420 and the system control unit 501.

The pixel memory 414 may further include a data transfer interface for transferring the pixel signal to the image processing unit 511 in accordance with the delivery request. The data transfer interface is connected to a data transfer line connected to the image processing unit 511. Such a data transfer line is formed by, for example, a part of the data bus, and a delivery request from the system control unit 501 to the drive control unit 420 is executed by address designation using the address bus.

In addition to the address specification method, the pixel signal transmission by the data transfer interface uses both the rising and falling edges of the clock signal used for synchronization of each circuit when transferring data, for example, to double data. It can also be a rate method. Further, by omitting a part of the procedure such as address designation, it is possible to adopt a burst transfer method in which data is transferred at once to increase the speed.

Further, for the transmission of the pixel signal by the data transfer interface, a bus method using a line connecting the control unit, the memory unit, and the input / output unit in parallel, a serial method of transferring data bit by bit in series, and the like are used. It can also be used in combination.

By using the data transfer interface as described above, the image processing unit 511 can receive only the required pixel signal. Thereby, for example, when forming a low-resolution image, the image processing can be completed at high speed. The drive unit 502, the row control unit 200 of FIG. 5, and the peripheral circuit 133 function as a reading unit that sequentially reads pixel signals from the pixels 150 across the plurality of unit blocks 131.

In the drive unit 502, the drive control unit 420 refers to the timing memory 430, converts an instruction received by the drive unit 502 from the system control unit 501 into a control signal that can be executed by each of the above control units, and delivers the instruction. The timing memory 430 is formed by a flash RAM or the like. In this way, the drive control unit 420 collectively controls the sensor control unit 441, the block control unit 442, the synchronization control unit 443, and the signal control unit 444.

Further, the illustrated drive unit 502 includes a correlation calculation unit 472, a control selection unit 474, and an image correction unit 476. The correlation calculation unit 472 detects the rolling shutter distortion included in the pixel signal based on the pixel signal read from the pixel memory 414. The control selection unit 474 selects each operation mode of the unit block 131 based on the correlation value calculated by the correlation calculation unit 472. The image correction unit corrects the rolling shutter distortion included in the pixel signal acquired from the unit block 131. The operations of the correlation calculation unit 472, the control selection unit 474, and the image correction unit 476 will be described later with reference to other figures.

FIG. 8 is a timing chart illustrating a rolling shutter mode, which is one operation mode of the unit block 131 in the image sensor 100. In the rolling shutter mode, in each of the unit blocks 131, the operation of selecting the pixel 150 and accumulating the pixel signal and the operation of reading the accumulated pixel signal are sequentially executed row by row.

In the illustrated example, the drive unit 502 first gives a drive signal to the wirings Rst_1 and Tx_1 for the pixel 150 in the first row via the row control unit 200. As a result, the charge accumulation of the pixel 150 in the first row of each unit block 131 is started.

Next, the drive unit 502 gives a drive signal to the wiring Self_1 for the pixel 150 in the first row via the row control unit 200 at the end of the accumulation period. As a result, the reading of the pixel signal of the pixel 150 in the first row is started. The read period is the time from when the pixel signal is read from each pixel 150 on the first line, processed by the analog / digital conversion circuit 202 and the CDS circuit 204, and sequentially written from the shift register 206 to the pixel memory 414. include.

Similarly, drive signals are given to the wirings Rst_2, Tx_2, and Ser_2 for the pixel 150 in the second row, and storage and reading are performed. The length of the accumulation period for the pixel 150 in the second row is the same as that for the pixel 150 in the first row, but the time is delayed by approximately the read period. From the third line to the Lth line, drive signals are given to the wirings Rst_l, Tx_l and Sel_l, and storage and reading are sequentially performed.

The order of the selected rows is the same for all of the plurality of unit blocks 131, and in the illustrated example, one row is selected from the −Y side row to the + Y side row for each of them. Will be done. Further, the timing of accumulating and reading the same row is synchronized among the plurality of unit blocks 131. However, at least one of the accumulation and reading timings of the same row may not be synchronized between the plurality of unit blocks 131.

FIG. 9 is a schematic view showing a subject 300 imaged in the rolling shutter mode using the image sensor 100. The subject 300 includes a linear image 301 that is vertical in the figure when the image light immediately before incident on the image sensor 100 is viewed from the + Z side. Further, as indicated by arrows in the figure, the linear image 301, at a velocity V _0, horizontally in the drawing, is moving in the -X direction.

FIG. 10 is a schematic diagram showing an image obtained when the subject 300 including the moving linear image 301 is imaged in the rolling shutter mode. For the sake of explanation, the subject 300 is drawn as a captured image in the orientation when it is output, displayed, or the like. Further, the subject 300 will be described as being imaged by four unit blocks 131 in the vertical and horizontal directions.

As described with reference to FIG. 8, in the rolling shutter mode, the timing at which the pixel signals are accumulated in one unit block 131 is sequentially shifted line by line. Therefore, the linear image 301 moving in the subject 300 is imaged at different positions depending on the timing of accumulating the pixel signals.

In each of the unit blocks 131, the timing of accumulating the pixel signal is different between the upper row in the figure and the lower row in the figure. Therefore, when the moving linear image 301 is imaged, the obtained captured image 311 is tilted even though the linear image 301 is vertical. That is, in the captured image 311, the longer the pixel signal accumulation period is in time, the more the image is imaged shifted in the + X direction, which is the moving direction, and rolling shutter distortion occurs for each unit block 131.

The above-mentioned displacement means a displacement in each of the four unit blocks 131 with the row in which the pixel signal is first accumulated as the initial position. In other words, it is considered that there is no deviation in the line in which the pixel signal is first accumulated in each unit block 131. Therefore, in the entire image sensor 100, as shown by the dotted line B in the figure, the captured image 311 is discontinuously displaced in the rows of the pixels 150 adjacent to each other at the boundary of the unit blocks 131 adjacent to each other. Such rolling shutter distortion becomes more prominent as the horizontal component _{of the moving speed V 0 of the linear image 301 is larger.}

FIG. 11 is a timing chart illustrating a global shutter mode, which is one operation mode of the unit block 131 in the image sensor 100. In the global shutter mode, all the pixels 150 included in the unit block 131 are simultaneously reset and then selected, and the pixel signal is accumulated. In the global shutter mode, the accumulated pixel signals may be sequentially read line by line as in the rolling shutter mode.

In the global shutter mode, since the timing of accumulating pixel signals in the unit block 131 is the same, rolling shutter distortion does not occur in the captured image 311. However, since all the pixels 150 are reset at the same time, a large current flows, and electric charges are likely to leak from each other in adjacent rows. Therefore, compared to the rolling shutter mode, noise is likely to be added to the captured image. It has been known.

FIG. 12 is a flow chart illustrating the operation of the control selection unit 474 in the drive unit 502. The control selection unit 474 initially operates the image sensor 100 in the rolling shutter mode (step S101). As a result, in the drive unit 502, the pixel memory 414 acquires the pixel signal (step S102).

When a moving object is included in the subject 300, rolling shutter distortion occurs in the captured image 311 obtained from the pixel signal of the image sensor 100. Therefore, at the boundary of the unit block 131, the captured images 311 are not continuous in the rows of adjacent pixels 150. Therefore, in the drive unit 502, the correlation calculation unit 472 calculates the correlation of the captured image 311 obtained from the rows of the pixels 150 adjacent to each other at the boundary of the unit block 131 for each unit block 131 (step S103).

Here, the control selection unit 474 compares the correlation value calculated by the correlation calculation unit 472 with a predetermined threshold value, and examines whether or not the correlation value is larger than the threshold value (step S104). When it is found that the correlation value is larger than the threshold value in any of the unit blocks 131 (step S104 :: YES), that is, it is obtained from the row of adjacent pixels at the boundary between the unit block 131 and the adjacent unit block 131. When the captured images are continuous or substantially continuous, the control selection unit 474 returns the control to step S101 and continues the imaging in the rolling shutter mode. In this way, in the unit block 131 that executes imaging in the rolling shutter mode, a high-quality pixel signal that is less affected by noise is output from the image sensor 100.

On the other hand, when the calculated correlation value is not larger than the threshold value in step S104 (step S104: NO), the control selection unit 474 switches the operation of the image sensor 100 to the global shutter mode (step S105). In the unit block 131 for executing the shooting in the global shutter mode in this way, the pixel signal forming the captured image 311 without the rolling shutter distortion is output from the image sensor.

The unit block 131 in which the control selection unit 474 has selected to operate in the rolling shutter mode by the above series of procedures is switched to the global shutter mode when the captured image 311 is distorted by the rolling shutter. .. However, in the unit block 131 once started to operate in the global shutter mode, it is not possible to detect whether or not a moving object is included in the subject 300.

Therefore, the control selection unit 474 monitors the duration of the global shutter mode for the unit block 131 that has selected to operate in the global shutter mode (step S106). As a result, the unit block 131 continues the global shutter mode until the duration of the global shutter mode reaches a predetermined threshold value (step S106: NO).

On the other hand, when the duration of the global shutter mode reaches the threshold value (step S106: YES), the control selection unit 474 returns the control to step S101 and returns the operation of the unit block 131 to the rolling shutter mode again. As a result, the control selection unit 474 can redetermine the operation of the unit block 131 according to the state of the X desired to know at that time.

The correlation value used as a judgment material in step S104 may be the difference between the rows of adjacent pixels 150 at the boundary of the unit block 131. Further, it may be a correlation value calculated by statistical processing such as residual sum of squares.

Further, the duration of the global shutter mode may be the duration of the mode, the number of imaging frames, or the like. Further, for example, the threshold value of the global shutter mode duration may be changed according to the magnitude of the detected rolling shutter distortion.

FIG. 13 is a schematic view showing a subject 300 in which moving linear images 302 and 303 and stationary linear images 304 coexist. The subject 300 includes a linear image 302 that moves at a high speed at a speed V ₁ , a linear image 303 that moves at a low _{speed at a speed V 2, and a stationary linear image 304.} The linear images 302, 303, and 304 form vertical straight lines when the image light immediately before incident on the image sensor 100 is viewed from the + Z side.

FIG. 14 shows a captured image before correction when the subject 300 including the linear images 302, 303, and 304 as described above is photographed in the rolling shutter mode in all the unit blocks 131. In the control procedure shown in FIG. 12, it corresponds to step S101. The captured images 312, 313, and 314 are drawn in the orientation when they are output or displayed. Further, it is assumed that the image sensor 100 is imaged by 12 unit blocks 131, each of which includes four pixels 150 in the vertical and horizontal directions.

Significant rolling shutter distortion occurs in the captured image 312 including the linear image 302 moving at high speed. Therefore, at the boundary of the unit blocks 131 arranged in the column direction, the correlation value of the rows of the adjacent pixels 150 becomes low. Therefore, the control selection unit 474 selects imaging in the global shutter mode for the unit block for capturing the captured image 312 including the linear image 302.

On the other hand, a slight rolling shutter distortion occurs in the captured image 313 including the linear image 303 moving at a low speed. Therefore, at the boundary of the unit blocks 131 arranged in the column direction, the correlation value of the rows of adjacent pixels 150 does not decrease. Therefore, the control selection unit 474 selects imaging in the rolling shutter mode with respect to the unit block for capturing the captured image 313 including the linear image 303. As a result, a slight rolling shutter distortion remains in the captured image 313, but the influence of noise is small.

Further, rolling shutter distortion does not occur in the captured image 314 including the stationary linear image 304. Therefore, at the boundary of the unit blocks 131 arranged in the column direction, the correlation value of the rows of adjacent pixels 150 does not decrease at all. Therefore, the control selection unit 474 selects imaging in the rolling shutter mode with respect to the unit block for capturing the captured image 314 including the linear image 304. As a result, the captured image 314 becomes a captured image that is less affected by noise.

FIG. 15 shows images 322, 323, and 324 of the subject 300 obtained by capturing images of some unit blocks 131 in the rolling shutter mode and imaging the other unit blocks in the global shutter mode by the control selection unit 474. It is a schematic diagram. As shown in the figure, in the captured image 322 of the linear image 302 moving at high speed, the occurrence of rolling shutter distortion is prevented. Further, in the other captured images 323 and 324, a high-quality image with less influence of noise can be obtained by the rolling shutter mode.

In this way, the image sensor 100 suppresses rolling shutter distortion and maintains image quality by determining and switching between the rolling shutter mode and the global shutter mode for each unit block 131. It is compatible. When the unit block 131 includes a special pixel such as a phase difference sensor for autofocus, it is preferable to exclude the unit block 131 from the control target of the control switching unit 474 and always operate in the global shutter mode.

In the example shown in FIG. 15, a slight rolling shutter distortion remains in the captured image 322. In the image sensor 100, the rolling shutter distortion generated in the image of the unit block 131 can be removed at the stage of the image sensor 100.

FIG. 16 shows another state of the subject 300 imaged by the image sensor 100. The illustrated subject 300 shows a state in which the image light immediately before being incident on the image sensor 100 is viewed from the + Z side.

The illustrated subject 300 includes linear images 305 and 307, each of which extends in the Y direction and moves in the X direction, and a stationary linear image 306. Further, the linear image 305 is moving at a higher speed than the linear image 307. However, in the illustrated example, it is assumed that the moving linear images 305 and 307 are moving at a speed in the range of being imaged in the rolling shutter mode.

FIG. 17 shows a captured image of the subject 300 shown in FIG. 16 taken by the image sensor 100 in the rolling shutter mode and before the rolling shutter distortion is corrected. In the figure, the subject 300 captured by the four unit blocks 131A, 131B, 131C, and 131D is shown in a direction in which the subject 300 is output, displayed, or the like as a captured image.

In the region imaged by the unit blocks 131A and 131B, rolling shutter distortion occurs in the captured image 315 corresponding to the linear image 305. Therefore, the captured image 315 is inclined at an angle. That is, the longer the accumulation period is in time, the larger the deviation in the + X direction, which is the moving direction, in the captured image 315.

Similarly, in the unit block 131D, the linear image 307 is distorted in the + X direction. However, the amount of distortion of the captured image 317 is small corresponding to the speed of the linear image 307 being smaller than the speed of the linear image 305. In the unit blocks 131C and 131D, the rolling shutter distortion does not occur in the captured image 316 corresponding to the stationary linear image 306.

Here, attention is paid to the row of the pixels 150 adjacent to each other at the boundary of the unit blocks 131A and 131B adjacent to each other in the same column. In the illustrated example, attention is paid to the tenth line of the unit block 131A and the first line of the unit block 131B shown by the dotted line B. Since the timings of the accumulation periods of these two rows are the most distant, a deviation occurs at the boundary between the unit blocks 131A and 131C even though the linear image 305 forming a continuous straight line is imaged.

As shown in FIG. 17, FIG. 18 is a flow chart showing a control procedure for correcting the rolling shutter distortion generated in the captured images 315 and 317 taken in the rolling shutter mode. FIG. 19 is a schematic view showing a captured image corrected by the procedure shown in FIG. The flowchart of FIG. 18 starts when the pixel signal corresponding to the uncorrected captured images 315, 316, and 317 shown in FIG. 17 is stored in the pixel memory 414.

When correcting the rolling shutter distortion, the drive unit 502 first reads the pixel signal of the boundary row in the unit blocks 131A, 131B, etc. adjacent to each other in the same column from the pixel memory 414 in the correlation calculation unit 472 (step S201). For example, in the case of a set of unit blocks 131A and 131B, the pixel signal on the 10th line of the unit block 131A and the pixel signal on the first line of the unit block 131B are read out.

Next, the correlation calculation unit 472 calculates the correlation value of the rows of adjacent pixels 150 at the boundary of the unit block 131 (step S202). It can be said that the magnitude of the correlation value is the degree of coincidence of the pixel signals in one column, and the higher the correlation value, the more the images in the adjacent rows match.

As an example of the correlation value, the correlation calculation unit 472 squares the difference between the pixel signals of the pixels in the same column in the 10th row of the unit block 131A and the 1st row of the unit block 131B, and sums them. Calculate the sum of squared differences. The smaller the sum of squares of the residuals, the greater the correlation. Hereinafter, an example in which the sum of squared residuals is used as the correlation value will be described.

The correlation calculation unit 472 determines whether or not the residual sum of squares is larger than the threshold value (step S203). When the magnitude of the residual sum of squares is equal to or less than the threshold value (step S203: NO), the flowchart in the unit blocks 131A, 131B, etc. adjacent to each other in the same column ends.

When the magnitude of the residual sum of squares is smaller than the threshold value, for example, in the example shown in FIG. 17, the 10th row on the unit block 131C side is later in time than the 1st row of the unit block 131D. It is presumed that the deviation of the row position is small despite the accumulation and reading. Therefore, in the unit blocks 131C and 131D adjacent to each other in the same column, the unit block 131C having the 10th row in which the accumulation and reading are performed later in time does not have or has the rolling shutter distortion. Is also presumed to be small.

Therefore, the unit block 131C is temporarily excluded from the correction target of FIG. However, the captured image 316 of the unit block C may also be affected by the adjacent unit blocks 131A in the same row, resulting in the correction block 132C.

When the magnitude of the sum of squares of the residuals is larger than the threshold value in the determination of step S203 (step S203: YES), the correlation calculation unit 472, on the other hand, in the 10th line of the unit block 131A and the 1st line of the unit block 131B. Is shifted by one pixel in the row direction with respect to the other (step S204), that is, the residual sum of squares is calculated for the pixels shifted by one column (step S205). The correlation calculation unit 472 repeats steps S204 and S205 for calculating the residual sum of squares by further shifting by one pixel until the number of pixels p in the column is reached (step S206: NO). In this case, the pixels are shifted in the + X direction and the −X direction, respectively.

The correlation calculation unit 472 specifies the number of pixels to be used for correction (step S207) after shifting by p pixels (step S206: YES). In this case, the correlation calculation unit 472 uses the number of pixel shifts when the smallest residual sum of squares is obtained from the plurality of residual sums of squares calculated by repeating steps S204 to S206 for correction. Let it be the number of pixels. When there are a plurality of sums of squares having the smallest residual squares, it is preferable to set the number of pixels to be smaller than the number of pixel shifts.

In the example shown in FIG. 19, when the 10th row of the unit block 131A is shifted in the −X direction by 4 pixels with respect to the 1st row of the unit block 131B, the residual sum of squares becomes the smallest. Therefore, the correlation calculation unit 472 specifies "4" as the number of pixels used for correction.

The image correction unit 476 corrects the pixel signal of the unit block 131A using the number of pixels specified in step S112 (step S208). Here, among the adjacent unit blocks 131A and 131B in the same column, the pixel signal of the unit block 131A having the 10th row in which the accumulation and reading are performed later in time is corrected.

In this case, the image correction unit 476 evenly allocates the deviation to each row in the unit block 131A so that the 10th row is shifted by 4 pixels, that is, 4 columns are shifted from the 1st row. In the example of FIG. 11, two adjacent rows are set as one set, and a deviation of one pixel is allocated to the adjacent sets.

As shown in FIG. 19, the image correction unit 476 fixes the 5th and 6th rows in the center of the unit block 131A with respect to the other unit block 131B and the like, and displays the pixel signals of the pixels in the other rows in columns. The correction block 132A is generated by replacing the pixel signal with the pixel signal of the shifted pixel. Similarly, the correction blocks 132B and 132D are generated by shifting the column positions of the respective rows of the unit blocks 131A and 132D.

The image correction unit 476 further corrects the pixel signals at the boundaries of the unit blocks 131A and 131C that are adjacent to each other in the same line. As shown in FIG. 13, when the correction blocks 132A and 132C are generated by correcting the adjacent unit blocks 131A and 131C in the same row, the area of the blank pixel 140 and the overlapping pixel 142 in which the pixels of each other overlap at the boundary thereof. Region arises. The image correction unit 476 allocates pixel signals from the periphery of the blank pixel 140 region. Further, the image correction unit 476 allocates the average value of the pixel signals of each pixel to the overlapping pixels 142.

As a result, the correction block 132A and the like are generated. Here, in the correction block 132C, the pixels are not replaced by the determination of the correlation in step S104 with respect to the unit block 131C, but the boundary thereof is corrected by the influence of the correction from the adjacent correction block 132A. The image correction unit 476 outputs a pixel signal of the correction block 132A or the like to the pixel memory 414 (step S209), and ends this flowchart. As described above, the rolling shutter distortion can be reduced with a simple configuration.

In step S202, the correlation between the boundary rows in the adjacent unit blocks 131A, 131B, etc. in the same column is calculated. Alternatively, the correlation between the first row and the last row in the same unit block 131A may be calculated.

Further, in each of steps S102 and 108, when the pixel signal has the luminance and color signals, the correlation may be calculated using the luminance signal. Further, instead of this, the correlation may be calculated for pixels of the same color, for example, pixels of green.

Furthermore, as the evaluation value of the correlation, the sum of the absolute values of the differences may be used instead of the sum of the residual squares. Further, instead of repeating for p pixels in step S110, a so-called mountain climbing method may be used in which the repetition is stopped when a minimum value appears, as compared with the calculation of the sum of squared residuals. ..

When the unit block 131 has a color filter such as a Bayer array, it is preferable to execute the above operation before performing the interpolation processing. Further, when the analog / digitally converted pixel signal is output as an image captured image converted into a predetermined format such as JPEG, the operation of FIG. 12 is after the analog / digital conversion, and the format is the same. It is preferably executed before converting to the format. Further, at least the correlation calculation unit 472 and the image correction unit 476 are preferably provided in the signal processing board 111.

As described above, in the image sensor 100, the rolling shutter distortion generated may be corrected for the unit block 131 photographed in the rolling shutter mode. However, in the image sensor 100, the unit block 131 in which the linear image 302 moving at high speed is imaged can be imaged in the global shutter mode in which rolling shutter distortion does not occur. Therefore, the processing load caused by the correction of the rolling shutter distortion is small.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The order of execution of each process such as operation, procedure, step, and step in the device, system, program, and method shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

10 Condenser, 60 Amplification Transistor, 61-1, 61-2 Selection Transistor, 59 Reset Transistor, 65 Ground Side Transistor, 78 1st Transistor, 63 2nd Transistor, 66 Amplification Transistor, 64 Transfer Transistor, 100 Imaging Element, 101 Micro Lens, 102 color filter, 103 passivation film, 104, 104-1, 104-2 photodiode, 105 transistor, 106 photoelectric conversion element layer, 107 wiring, 108 wiring layer, 109 bump, 110 TSV, 111 signal processing board, 112 Memory board, 113 light receiving board, 131 unit block, 132 correction block, 133 peripheral circuit, 140 blank pixels, 142 overlapping pixels, 150 pixels, 170 columns transmission line, 172 columns bus line, 200 rows control unit, 202 analog / digital conversion Circuit, 204 CDS circuit, 206 shift register, 300 subject, 301, 302, 303, 304, 305, 306, 307 Linear image, 311, 312, 313, 314, 315, 316, 317, 311, 322, 323, 324 Captured image, 325, 327 corrected image, 414 pixel memory, 418 I / F circuit, 420 drive control unit, 430 timing memory, 441 sensor control unit, 442 block control unit, 443 synchronization control unit, 444 signal control unit, 472 correlation Calculation unit, 474 control selection unit, 476 image correction unit, 500 imaging device, 501 system control unit, 502 drive unit, 503 metering unit, 504 work memory, 505 recording unit, 506 display unit, 511 image processing unit, 512 arithmetic unit 520 Imaging lens

Claims (10)

With multiple pixels
Among the plurality of pixels, the global electronic shutter operation and the rolling electronic shutter operation are performed by the first row control unit for the plurality of first pixels arranged side by side in the row direction and the column direction in the first region. It is controlled so that one of the electronic shutter operations is performed , and among the plurality of pixels, a plurality of second pixels arranged side by side in the row direction and the column direction in the second region. , The second row control unit is provided with a control unit that controls the electronic shutter operation of either the global electronic shutter operation or the rolling electronic shutter operation.
The plurality of pixels are arranged on a light receiving substrate on which light is incident, respectively.
The first row control unit and the second row control unit are arranged on a signal processing board different from the light receiving board, respectively.
When the control unit causes the plurality of first pixels to perform the global electronic shutter operation, the period during which the signal is read from the first pixel by the global electronic shutter operation is a predetermined period. When it exceeds, the rolling electronic shutter operation is controlled to be performed on the plurality of first pixels, and the global electronic shutter operation is performed on the plurality of second pixels. An image pickup device that controls the plurality of second pixels to perform the rolling electronic shutter operation when the period during which a signal is read from the second pixel exceeds a predetermined period. With multiple pixels
Among the plurality of pixels, the global electronic shutter operation and the rolling electronic shutter operation are performed by the first row control unit for the plurality of first pixels arranged side by side in the row direction and the column direction in the first region. It is controlled so that one of the electronic shutter operations is performed , and among the plurality of pixels, a plurality of second pixels arranged side by side in the row direction and the column direction in the second region. , The second row control unit is provided with a control unit that controls the electronic shutter operation of either the global electronic shutter operation or the rolling electronic shutter operation.
The plurality of pixels are arranged on a light receiving substrate on which light is incident, respectively.
The first row control unit and the second row control unit are arranged on a signal processing board different from the light receiving board, respectively.
When the control unit causes the plurality of first pixels to perform the global electronic shutter operation, the number of times the signal is read from the first pixel by the global electronic shutter operation is a predetermined number of times. When it exceeds, the rolling electronic shutter operation is controlled to be performed on the plurality of first pixels, and the global electronic shutter operation is performed on the plurality of second pixels. An image pickup device that controls the plurality of second pixels to perform the rolling electronic shutter operation when the number of times a signal is read from the second pixel exceeds a predetermined number of times. In the image pickup device according to claim 1 or 2.
The control unit is an image pickup device arranged on a signal processing board different from the light receiving board. In the image pickup device according to claim 3,
A conversion circuit for converting a signal output from the first pixel into a digital signal is provided.
The conversion circuit is an image pickup device arranged on the signal processing substrate. In the image pickup device according to claim 4,
The light receiving substrate is an image pickup element laminated by the signal processing substrate. The image sensor according to any one of claims 1 to 5.
The first pixel is an image pickup element having a photoelectric conversion unit that converts light into electric charges and a transistor that is connected to the control unit and for transferring charges of the photoelectric conversion unit. In the image pickup device according to claim 6,
An image pickup device including wiring for outputting a control signal from the control unit to the transistor. An image pickup apparatus comprising the image pickup device according to any one of claims 1 to 7. In the imaging device according to claim 8,
An image pickup device including an image processing unit connected to the image pickup device and generating image data. In the imaging device according to claim 9,
An imaging device including a display unit for displaying an image of image data generated by the image processing unit.

Images