JP7686438B2 - Image pickup device, image pickup apparatus, monitoring apparatus, and image pickup device control method - Google Patents

Description

The present invention relates to an imaging element, an imaging device, a monitoring device, and a method for controlling an imaging element.

An imaging device has been proposed that detects a moving subject from an image captured by an imaging element and changes the operation of the imaging element. Patent Document 1 proposes an imaging device that changes the operation settings of the imaging element based on the presence or absence of a moving subject in an image. The imaging device described in Patent Document 1 has a frame memory that stores captured pixel signals in chronological order, and is configured to compare images stored in the frame memory and have a state determination unit determine the operational state of the pixels based on the presence or absence of a moving subject in the image. The imaging device determines the operation of the image sensor when acquiring images from the next frame onwards based on the presence or absence of a moving subject in the image.

International Publication No. WO 2018/021034

In order to detect a moving subject, the imaging device of Patent Document 1 is provided with a large frame memory that stores multiple images obtained from the imaging element, and compares the images between frames. Therefore, there are issues with the addition of a large frame memory, such as increased costs and increased power consumption. In addition, since image judgment is performed after one image is obtained from the imaging element, there is an issue that it is difficult to improve responsiveness to images.

In view of the above problems, the present invention aims to provide an imaging device that can easily and quickly capture images when an event occurs.

The first aspect of the present invention is
An imaging element having a plurality of pixels,
The accumulated charges of the plurality of pixels are periodically reset;
In at least a portion of the plurality of pixels,
performing a first readout process for reading out accumulated charges due to the first exposure;
performing a first signal process to detect whether or not an event has occurred based on a first pixel signal read out by the first readout process;
a determination is made according to a result of the first signal processing as to whether or not to perform a second readout process for reading out accumulated charges due to a second exposure and a second signal processing for correcting a second pixel signal read out by the second readout process so that the second pixel signal is suitable as an image ;
the first readout process, the first signal processing, and the second readout process are performed within one period of a reset that is performed periodically;
It is an image sensor.

A second aspect of the present invention is
A method for controlling an image sensor including a plurality of pixels each including a photoelectric conversion unit, a readout unit that reads out charges accumulated in the photoelectric conversion unit, and a reset unit that resets the photoelectric conversion unit, and a signal processing unit that performs signal processing on pixel signals output from the plurality of pixels, comprising:
periodically resetting the photoelectric conversion units in the plurality of pixels;
performing a first readout process for reading out accumulated charges due to a first exposure;
performing a first signal processing step of detecting whether or not an event has occurred based on a first pixel signal read out by the first readout processing;
a step of determining whether or not to perform a second readout process for reading out accumulated charges due to a second exposure and a second signal process for correcting a second pixel signal read out by the second readout process so that the second pixel signal is suitable as an image , according to a result of the first signal processing;
Including,
the first readout process, the first signal processing, and the second readout process are performed within one period of a reset that is performed periodically;
This is a method for controlling an image sensor.

The present invention makes it possible to easily and quickly obtain images when an event occurs.

FIG. 2 is a schematic diagram illustrating a configuration of an image sensor according to an embodiment. FIG. 2 is a schematic diagram illustrating a configuration of an image sensor according to an embodiment. FIG. 2 is a schematic diagram illustrating a configuration of an image sensor according to an embodiment. FIG. 2 is a schematic diagram illustrating a configuration of an image sensor according to an embodiment. FIG. 2 is a schematic diagram illustrating a configuration of an image sensor according to an embodiment. FIG. 4 is a schematic diagram illustrating a processing flow of an image sensor according to an embodiment. 5A and 5B are schematic diagrams illustrating processing timing of an image sensor according to an embodiment. 2 is a schematic diagram illustrating the configuration of an image sensor according to an embodiment in more detail. FIG. 2 is a schematic diagram illustrating the configuration of an image sensor according to an embodiment in more detail. FIG. 2 is a schematic diagram illustrating the configuration of an image sensor according to an embodiment in more detail. FIG. 2 is a schematic diagram illustrating the configuration of an image sensor according to an embodiment in more detail. FIG. 5A and 5B are schematic diagrams illustrating processing timing of an image sensor according to an embodiment. FIG. 11 is a schematic diagram illustrating a processing flow of an image sensor according to a second embodiment. 13A and 13B are schematic diagrams illustrating an imaging element according to a third embodiment. 13A and 13B are schematic diagrams illustrating processing timing of an image sensor according to a fourth embodiment. 13A to 13C are schematic diagrams illustrating processing of an image sensor according to a fifth embodiment. 13 is a schematic diagram illustrating an imaging element according to a sixth embodiment. FIG. 13 is a schematic diagram illustrating an imaging element according to a seventh embodiment. FIG. FIG. 13 is a schematic diagram illustrating an imaging element according to an eighth embodiment. 13A and 13B are schematic diagrams illustrating imaging devices according to ninth and tenth embodiments.

The following describes embodiments of the present invention with reference to the drawings. However, the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the present invention.

(First embodiment)
Fig. 1A is a schematic diagram illustrating the configuration of an image sensor 100 provided in an image sensor according to an embodiment of the present invention. The image sensor 100 includes a photoelectric conversion unit 110, a reset unit 120, a signal readout unit 130, a signal processing unit 140, and a signal output unit 150. The signal readout unit 130 includes an amplifier 131 and a pixel selection unit 132. In Fig. 1A, reference numeral 200 denotes incident light, 201 denotes a charge signal, 202 denotes a light detection signal, 203 denotes a pixel output after signal processing, 204 denotes an image output signal, and 205 denotes a photodiode charge reset signal.

In this embodiment, a periodic charge reset process is performed on the photoelectric conversion unit 110. During one period, the signal processing unit 140 performs a first readout process to read out accumulated charges due to a first exposure, and a first signal processing based on a first pixel signal read out by the first readout process. The signal processing unit 140 further determines whether or not to perform a second readout process to read out accumulated charges due to a second exposure, and a second signal processing based on a second pixel signal read out by the second readout process, depending on the result of the first signal processing. The first readout process, the first signal processing, the second readout process, and the second signal processing are performed within one period of the periodic reset process.

In FIG. 1A, the photoelectric conversion unit 110 is composed of, for example, a photodiode. The photodiode can generate and accumulate electric charges according to the intensity and time of received light. In this specification, the light receiving unit of the photodiode is called a pixel. One pixel includes the photoelectric conversion unit 110, a signal readout unit 130 that reads out the electric charges accumulated in the photoelectric conversion unit 110, and a reset unit 120 that resets the photoelectric conversion unit 110. The image sensor 100 has a configuration in which multiple pixels are arranged one-dimensionally or two-dimensionally. The electric charges generated in the photoelectric conversion unit (photodiode) 110 are converted into a voltage signal in the signal readout unit 130, transferred, and output to the signal processing unit 140.

The signal readout unit 130 includes an amplifier 131 for each pixel that converts the charge into a voltage. The output from the amplifier 131 for each pixel is connected to and output from a pixel selection unit 132. The signal readout unit 130 is driven based on a control signal from a control circuit (not shown).

The signal processing unit 140 performs signal processing on pixel signals output from multiple pixels. The signal processing unit 140 has, first, a function of detecting the occurrence of an event based on signals from each pixel (first signal processing function), and, second, a function of correcting the signals received from each pixel and outputting the signals to the signal output unit 150 (second signal processing function). The detailed operation of the signal processing unit 140 will be described with reference to Figures 2 and 3.

The signal output unit 150 outputs the signals from all pixels received from the signal processing unit 140 all at once to the outside. In this specification, the output signal containing information from all pixels is called an image signal, and one set of image signals is called a frame. The image signal is used outside the image sensor 100 to modify, display, or record one frame of captured image. At this time, outside the image sensor 100, the signal positions are generally rearranged and the brightness and color are adjusted.

The signal output unit 150 also repeatedly outputs the image output signal for each frame, usually at a constant frequency fout (=1/period T). This frequency fout is called the frame rate, and is an index showing how many sets of frame images are output per second. The unit of frame rate is fps (frames per second). The specific value of the frame rate varies depending on the number of pixels in the image sensor, but is generally several fps to several hundred fps.

The accumulated charge in the photoelectric conversion unit 110 is read by the signal readout unit 130, but if the accumulated charge is not removed after reading, unnecessary signals will be mixed into the image signal, degrading the quality (image quality) of the image signal related to the next readout. For this reason, the reset unit 120 removes the charge held by the photoelectric conversion unit 110 (resets the charge of the photoelectric conversion unit 110). In order to prevent degradation of image quality, it is desirable to perform this charge reset process of the photoelectric conversion unit 110 for all pixels in a frame period before charge accumulation in the pixels.

In FIG. 1A, for the sake of simplicity, the imaging element 100 has only one pixel. However, as shown in FIG. 1B, the imaging element 100 may have a plurality of pixels and a signal transfer unit 133 for transferring the light detection signals of the plurality of pixels. The signal transfer unit 133 is a means for reading and transferring the light detection signals, and a transfer method for a CMOS sensor is used. Specifically, the imaging element 100 has a plurality of pixels arranged two-dimensionally, a signal transfer unit 133 is arranged for each pixel in the row (horizontal) direction, and the signal transfer unit 133 is shared by the pixel selection unit 132 in the column (vertical) direction. The signal transfer unit 133 is connected to the signal processing unit 140, and an output signal from a pixel selected by the pixel selection unit 132 is transmitted to the signal processing unit 140 via the signal transfer unit 133. By sequentially selecting a plurality of pixels in chronological order, the signals from the plurality of pixels can be sequentially extracted using the common signal transfer unit 133.

A specific example of the configuration of the photoelectric conversion unit 110 and the signal readout unit 130 will be described with reference to Figures 4A and 4D. For simplicity, only one pixel 101 is shown in Figure 4A, but in reality, multiple pixels 101 are provided for one transfer signal line 441, as shown in Figure 4D. Also, Figure 4D shows the configuration for one pixel column, and the image sensor 100 has multiple columns of the configuration shown in Figure 4D. Note that all of the transistors described in this specification are MOS transistors.

In FIG. 4A, a photodiode 401 is used as the photoelectric conversion unit 110, and the cathode terminal is connected to GND. A reset transistor 431 is arranged between the positive power supply voltage VDD and the photodiode 401 as the reset unit 120. A drive transistor 411 and a load transistor 412 are arranged in series between the positive power supply VDD and the negative power supply VSS as the charge-voltage conversion amplifier 131. The photodiode 401 is connected to the gate input GATE of the drive transistor 411. The positive power supply voltage VDD, the drive transistor 411, the load transistor 412, and the negative power supply VSS constitute a source follower amplifier 410. The source follower amplifier 410 has high input impedance and low output impedance, and works as a buffer amplifier that converts the charge accumulated in the input side element into a voltage and transmits it to the subsequent circuit.

A selection transistor 420 is used as the pixel selection unit 132, and is disposed between the drive transistor 411 and the load transistor 412. By turning on the selection transistor 420, a source follower amplifier between the positive power supply VDD and the negative power supply VSS operates, and VOUT is generated. The signal transfer unit 133 is a transfer signal line 441, and is connected to the VOUT terminal of the source follower amplifier. As shown in FIG. 4D, all the VOUT terminals of the pixels 101 that share this signal transfer unit 133 are connected to the same transfer signal line 441. For multiple pixels connected to the same transfer signal line 441, only the selection transistor 420 of one pixel is selected at the same time. By sequentially switching the selected transistor, all the outputs from each pixel can be read out in chronological order. Note that the load transistor 412 of the source follower amplifier does not need to be provided for each pixel, and only one is required to be disposed for each transfer signal line 441. The imaging element may also have multiple pixel columns and transfer signal lines 441 as shown in FIG. 4D.

The reset unit 120 is composed of a reset transistor 431 arranged between the positive power supply VDD and the photodiode 401. Before starting exposure, the reset transistor 431 is turned ON to remove the charge accumulated in the photodiode 401.

Another example of the configuration of the photoelectric conversion unit 110 and the signal readout unit 130 will be described with reference to Fig. 4B. Fig. 4B shows a configuration in which a diffusion capacitance FD402 and an FD transfer transistor 421 are added to the configuration shown in Fig. 4A. The diffusion capacitance FD402 is disposed between the gate input GATE of the driving transistor 411 and the GND terminal. Furthermore, the selection transistor 420 is inserted in a wiring portion between the cathode terminal of the photodiode 401 and the gate input GATE of the driving transistor 411. In the configuration of Fig. 4B, the FD transfer transistor 421 is normally turned OFF during exposure, and is turned ON at the end of exposure to transfer the charge accumulated in the photodiode 401 to the diffusion capacitance FD402, so that the charge can be converted into a voltage signal by the driving transistor 411.

In addition, in FIG. 4B, the position where the reset transistor 431 is arranged is different from that in FIG. 4A, and it is arranged between the positive power supply VDD and the gate input GATE of the drive transistor 411. Therefore, by turning on the reset transistor 431, it is possible to reset the accumulated charge of the photodiode 401 and the charge accumulated in the diffusion capacitance FD402.

Another configuration will be described with reference to FIG. 4C. FIG. 4C shows an example of a pixel configuration having a global electronic shutter (GS) function that can transfer all pixels at once. In addition to the configuration of FIG. 4B, this configuration adds one storage capacitor section 403 and two transistors (a second reset transistor 432 and a GS transfer transistor 422). For the sake of explanation, the position of the reset transistor 431 has been changed between FIG. 4C and FIG. 4B.

When the GS transfer transistor 422 is in the ON state, it transfers the charge accumulated in the photodiode 401 to the holding capacitance section 403. When the FD transfer transistor 421 is in the ON state, it transfers the charge in the holding capacitance section 403 to the FD capacitance section 402. In addition, the accumulated charge in the photodiode 401 can be reset by turning on the reset transistor 431.

Next, the processing flow performed by the image sensor 100 in this embodiment will be described with reference to FIG. 2. The following processing is performed based on a control signal from a control circuit, but for simplicity, reference to the control circuit and control signal will be omitted. Also, for simplicity, the processing flow will be described for one pixel.

The process in FIG. 2 begins by receiving an image capture instruction in step 300. In step 310, the reset unit 120 is driven to reset the charge accumulated in the photoelectric conversion unit 110. The photodiode 110, whose accumulated charge has now been reset, begins receiving light and accumulating charge. When a predetermined first time T1 has elapsed since the accumulated charge was reset, the pixel signal readout process in step 320 is performed.

In step 320, the signal readout unit 130 reads out a first pixel signal corresponding to the accumulated charge of the pixel (first readout process). In this specification, the period from the start of light reception to the readout of the pixel signal is called the exposure period. Note that as soon as the reading of the pixel output in step 320 is completed, the pixel starts receiving light again. In other words, in this embodiment, the exposure is temporarily stopped for a very short time by the first readout process, but exposure continues during the first signal processing period described below. This prevents the image information to be acquired from being interrupted by the first signal processing, making it possible to acquire image information with higher real-time accuracy.

In step 330, the signal processor 140 detects whether an event has occurred based on the pixel output signals simultaneously read out in step 320 (first signal processing). The signal processor 140 determines that an event has occurred when a moving subject in an image satisfies a predetermined condition. Examples of the predetermined condition include a moving subject suddenly stopping, a still subject suddenly starting to move, or a subject moving in a manner that differs from an allowable or expected movement.

In step 340, the signal processing unit 140 branches the process depending on whether an event has been detected.

If an event is detected (340-YES), the process proceeds to step 350. In step 350, after a predetermined second time T2 has elapsed since step 320, the signal readout unit 130 reads out a second pixel signal corresponding to the accumulated charge of the pixel (second readout). During this second readout period, the signal exposed (accumulated) during the time period T2 from the first readout timing to the second readout timing is extracted. In step 360, the signal processing unit 140 corrects the second pixel signal (second signal processing). In step 370, the signal processing unit 140 outputs the corrected frame image.

On the other hand, if no event is detected (340-NO), the process proceeds to step 371. In step 371, the second readout by the signal readout unit 130 and the second signal processing by the signal processing unit 140 are not performed. Furthermore, an invalid frame image is generated in the invalid frame image generation unit 151 of the signal output unit 150 and output to the outside. Here, an "invalid frame image" is any information different from the second signal obtained when the second readout is assumed to have been performed, and is most simply meaningless data. Examples of meaningless data include an image in which all pixel values are filled with zero values or other specific values, and an image having a specific pattern. In addition, as a form for outputting an invalid frame image, a form for outputting an entire image with a long update time (dropped frames) or a form for outputting a rough image obtained in the first readout can also be adopted.

In step 380, it is determined whether the next frame is to be captured, and if so, the process returns to step 310; if not, the process ends (390).

When repeated shooting is performed, the reset process (step 310) of the accumulated charge in the photoelectric conversion unit 110 by the reset unit 120 is set at a predetermined cycle in each pixel, and the frequency is the same as the frame rate. On the other hand, the frequency at which the image signal is output from the signal output unit 150 (steps 370 and 371) is also set to be the same as the frame rate. Note that the reset cycle is the same for each pixel, but the reset timing may be the same for all pixels, or may be different for each pixel or each pixel row.

In this specification, the exposure and readout operations are collectively referred to as imaging, the first exposure and first readout are collectively referred to as first imaging, and the second exposure and second readout are collectively referred to as second imaging. The first imaging can be referred to as imaging (exposure and readout) for acquiring pixel information for event detection, and the second imaging can be referred to as imaging (exposure and readout) for acquiring pixel information for generating an output image. In addition, the set of pixel data obtained in the first imaging is referred to as first subframe pixel data, and the set of pixel data obtained in the second imaging is referred to as second subframe pixel data.

The operation timing of the image sensor 100 according to this embodiment will be explained using FIG. 3. To simplify the explanation, a timing chart will be explained for one pixel.

First, in order to keep the accumulated charge in the photoelectric conversion unit 110 constant, a photodiode charge reset signal 205 is generated as shown in FIG. 3(a). This photodiode charge reset signal 205 is a signal that repeats at a constant cycle, and as described above, resets all pixels periodically at the same cycle as the frame cycle.

During one reset cycle, the first image capture (first exposure and first readout) is performed at each pixel.

As shown in FIG. 3B, the photoelectric conversion unit 110 of the reset pixel receives light and starts accumulating electric charge (start of exposure period). This exposure period continues until the next first readout signal is generated.

As shown in FIG. 3C, when the first readout signal is input to the signal readout unit 130, the signal readout unit 130 reads out the accumulated charge from the photoelectric conversion unit 110. The readout signal is called the first pixel signal. The details of the operation at this time differ depending on whether the configuration of the image sensor 100 is as shown in FIG. 4A to FIG. 4C, but ultimately the charge signal is output as a voltage signal.

As shown in (d) of FIG. 3, the signal processing unit 140 performs event detection signal processing (first signal processing) on the first pixel signal read out in the first readout process. Simultaneously with the start of this signal processing, as shown in (f) of FIG. 3, the photoelectric conversion unit 110 starts a second exposure period, and the exposure period continues until the second readout signal is generated ((g) of FIG. 3).

(e) in FIG. 3 shows the result of the first signal processing, in which a signal is output according to whether an event has been detected or not (340-YES or 340-NO). In this embodiment, if an event has been detected, a high level signal is output, and if no event has been detected, a low level signal is output.

Depending on the result of the first signal processing, the signal processing unit 140 judges whether or not to perform a second readout process for reading out the accumulated charge related to the second exposure and a second signal processing based on the second pixel signal read out by the second readout process. Specifically, when the occurrence of a predetermined event is detected in the first signal processing (340-YES), the signal processing unit 140 judges to perform the second readout process and the second signal processing. On the other hand, when the occurrence of a predetermined event is not detected in the first signal processing (340-NO), the signal processing unit 140 judges not to perform the second readout process and the second signal processing. When the second readout process and the second signal processing are not performed, the invalid frame image generating unit 151 included in the signal output unit 150 generates an invalid image signal and outputs the invalid frame image signal (DUM DATA in FIG. 3(i)) from the signal output unit 150. Hereinafter, a case where the second readout process and the second signal processing are performed will be described.

As shown in (g) of FIG. 3, when the occurrence of a predetermined event is detected (340-YES), the second readout signal is input to the signal readout unit 130, and the signal readout unit 130 performs a second readout process to read out the accumulated charge from the photoelectric conversion unit 110. The readout signal is called a second pixel signal. As shown in (h) of FIG. 3, the signal processing unit 140 performs a correction (adjustment) process (second signal processing) to make the second pixel signal suitable as an image. The second signal processing is a process to generate an image signal for output from the image sensor 100 based on the second pixel signal. In (i) of FIG. 3, the adjusted image signal is output from the signal output unit 150 to the outside as a single image output (OUT DATA section in (h) of FIG. 3).

As shown in Fig. 3, the period T for outputting the image signal and the period T for the reset signal are the same. As shown in Fig. 3, once the second readout is complete, the pixel information itself has already been acquired, so the reset signal can be generated even during the second signal processing or image output. This reduces wasted time and makes it possible to minimize the period T for outputting the image.

In this embodiment, examples of predetermined events detected by the signal processing unit 140 include a still subject starting to move and a moving subject suddenly becoming still.

An image sensor that detects the detection of a moving subject as an event can be used to monitor suspicious individuals or intruders. In other words, only when a moving subject such as a suspicious individual or intruder is detected by the first readout process and the first signal processing (event detection process), a second readout process and a second signal processing are performed to output an image of the moment when the moving subject is detected.

In addition, an image sensor that detects the sudden stopping of a moving subject as an event can be used for traffic monitoring. For example, it can detect when a car suddenly stops due to a traffic accident and output an image of the exact moment that this is detected.

For example, the occurrence of an event can be easily detected based on the change between pixel data acquired a short time ago (a predetermined number of frames ago) and the newly acquired pixel data. The event detection function of this embodiment is not limited to the above functions and methods, and other functions and methods can be adopted as long as the detection is based on information obtained by shooting. According to this embodiment, regardless of the specific content of the event detection function, the event can be automatically detected only by the image sensor 100, and an image of the moment when the event occurs can be output in a timely manner. In addition, the occurrence of an event can be detected and an image of only that moment can be output. Alternatively, when the occurrence of an event is detected, the second readout process and the second signal process can be performed continuously for a predetermined frame period, and an image can be output. By continuing the image output for a predetermined frame period, it is possible to continue shooting the situation after the occurrence of the event.

Note that FIG. 3 illustrates a reset performed only once during one frame period to simultaneously reset all pixels. However, the present invention is not limited to this configuration, and a sub-reset may be performed one or more times between the completion of the first readout and the start of exposure. In this specification, the driving of the reset unit 120 performed before the start of the first exposure is called a reset (or a main reset), and the driving of the reset unit 120 performed before the start of the second exposure is called a sub-reset, and the two are considered to be clearly separate operations or processes.

The reset period is the same for all pixels, but the timing of the reset may be simultaneous for all pixels (global shutter method) or different for each pixel row (rolling shutter method).

As described above, in this embodiment, at intervals when the photoelectric conversion unit 110 performs periodic charge reset processing, a first signal is read out from the signal readout unit 130, and the first signal processing is performed in the signal processing unit 140. Then, based on the result of the first signal processing, it is determined whether or not to perform a second signal readout. Therefore, according to this embodiment, it is possible to provide an imaging element that can capture images without delay when an event occurs, without performing high-load signal processing or image transfer externally.

In the explanation of Figures 2 and 3, for simplicity, only one pixel is explained. If the image sensor has multiple pixels, so long as there are as many signal transfer units 133 as there are pixels, it can perform substantially the same operations as those explained in Figures 2 and 3. However, in many cases, the signal transfer unit 133 is shared by multiple pixels, so the readout timing is shifted slightly during readout.

In addition, in this embodiment, the first readout process, the first signal processing, the second readout process, and the second signal processing are performed within one periodic reset cycle, but this is not limited to this. For example, the first readout process, the first signal processing, and the second readout process may be performed within one reset cycle, and the second signal processing may be performed in the next cycle or the cycle after the next cycle.

In addition, in this embodiment, the signal processing unit 140 receives the event detection result (340-YES, 340-NO) of the signal processing (first signal processing) and does not perform the second readout or the second signal processing. However, the present invention is not limited to this. Even when the result is 340-NO, adopting a configuration that performs the second readout or the second signal processing will result in extra power consumption, but if the increase in power consumption is not a problem in practical use, these configurations may be adopted. In this case, since the signal output unit 150 has an invalid frame image generation unit 151, it is possible to perform the same operation as the present invention.

When the circuit configuration described in FIG. 4A or FIG. 4B is adopted, the timing can be described as shown in FIG. 5. In FIG. 5, 501 is a reset signal (main), 502 is a first exposure period, and 503 is a second exposure period. In FIG. 5, in order to simplify the drawing, only the timing for 8 pixels is shown. Also, due to space in the drawing, only 501, 502, and 503 are shown, but the other signals shown in FIG. 3 also operate at slightly shifted timings. Here, Tread1 is the first readout period, and Tcal1 represents the period required for the first signal processing. By adopting such a configuration, the degree to which the first readout affects the exposure can be minimized, and the exposure period can be extended. Also, since the second exposure continues during the first signal processing period, there is no need to stop the exposure. Also, when the circuit configuration described in FIG. 4C is adopted, the reset timing, i.e., the exposure timing, can be made the same for all pixels.

In addition, in FIG. 1B, the signal transfer unit 133 is directly connected to the signal processing unit 140, but as shown in FIG. 1C, a configuration can also be made in which a digital signal conversion unit 134 is provided between the signal transfer unit 133 and the signal processing unit 140. The digital signal conversion unit 134 can convert the analog output signal from the amplifier 131 of each pixel into a digital signal 207. This allows the digital signal to be directly input to the signal processing unit 140, making it easy to process signals using the digital signal.

In addition, in this embodiment, the signal processing unit 140 and the signal output unit 150 are provided inside the image sensor 100, but the signal processing unit 140 and the signal output unit 150 may be provided outside the image sensor 100.

Also, as shown in FIG. 1D, the signal readout unit 130 of each pixel includes a digital signal conversion unit 134 that digitizes the output signal of the amplifier 131, and can be configured to transmit a digital signal 207 to the signal transfer unit 133.

Furthermore, as shown in FIG. 1E, an amplifier 135 can be disposed between the signal transfer unit 133 and the signal processing unit 140, and the pixel signal can be amplified as a signal 208 and transmitted, thereby enabling the configuration to improve the signal-to-noise ratio.

Second Embodiment
The second embodiment is different from the first embodiment in that an image is output from the signal output unit 150 using pixel signals (image information) obtained in the first imaging, but is otherwise the same as the first embodiment. The following mainly describes the differences from the first embodiment.

Figure 6 is a flow diagram for explaining the operation of this embodiment. The operation of this embodiment is similar to that of the first embodiment (Figure 2) except that the content of step 360 has been changed. In Figure 6, dotted line 321 indicates the flow of pixel signals obtained by the first imaging, and dotted line 322 indicates the flow of pixel signals obtained by the second imaging.

In this embodiment, in the pixel signal adjustment process (step 365) after the second readout process (step 350), in addition to the pixel signal 322 during the second imaging, the pixel signal during the first imaging for determining an event is used to output an image from the signal output unit 150. Specifically, as shown in FIG. 6, in step 365, the signal processing unit 140 generates a frame image based on the pixel signal 321 from the first readout (step 320) and the pixel signal 322 from the second readout (step 350). The frame image is generated, for example, by combining the pixel signal 321 and the pixel signal 322. In step 370, the signal processing unit 140 outputs the generated frame image.

In this embodiment, the pixel signal obtained by the first readout used for event detection is also used to generate the frame image, so even if event detection is performed, the total exposure time for shooting is hardly reduced. Therefore, according to this embodiment, the effects of the first embodiment can be obtained, and a frame image with higher image quality can be output.

Third Embodiment
The third embodiment differs from the first embodiment in that the first readout is performed only for some pixels, and is otherwise the same as the first embodiment. Note that this embodiment may be implemented in combination with the second embodiment.

In this embodiment, the multiple pixels 600 are divided into a first pixel group and a second pixel group, and the first image is captured using only the first pixel group. Hereinafter, the pixels belonging to the first pixel group are also referred to as first pixels, and the pixels belonging to the second pixel group are also referred to as second pixels.

Fig. 7A is a schematic diagram for explaining an example of the pixel configuration of this embodiment. As shown in Fig. 7A, a first pixel group 601 and a second pixel group 602 are arranged in row (horizontal) units in the image sensor. That is, the first pixels and the second pixels are arranged in alternate rows, and when viewed in the column direction, the first pixels and the second pixels are arranged alternately.

The operation of this embodiment is basically the same as that of the first embodiment (Figure 2). However, in the first readout process of step 320, only pixels belonging to the first pixel group 601 are read out, and pixels belonging to the second pixel group 602 are not read out. The event occurrence detection process of step 330 is performed based on pixel signals obtained from the first pixel group 601. After an event occurrence is detected, the processes from step 350 onwards are performed on both the first pixel group 601 and the second pixel group 602.

As described above, according to this embodiment, since all pixel signals are not read out for event detection, but only a portion of the pixels are thinned out and read out, the transfer time of the pixel signals can be shortened. Therefore, the reduction in exposure time during the second image capture associated with the implementation of event detection can be minimized. In addition, since the number of pixels used in the event detection process (first signal processing) is limited, the load of the signal processing can be minimized, and it is possible to achieve miniaturization and reduced power consumption.

Furthermore, since the pixels belonging to the first pixel group 601 are scattered all over the screen, it is possible to detect an event over the entire screen. Here, in detecting an event, sufficient information can often be obtained even with pixel signals that have been thinned out, so this embodiment can be widely applied. It is desirable that the number of pixels belonging to the first pixel group 601 is smaller than the number of pixels belonging to the second pixel group 602. Furthermore, in consideration of the processing load and detection efficiency, it is desirable that the ratio of the number of pixels belonging to the first pixel group 601 to the total number of pixels is about several tens of percent to several percent.

In addition, it is necessary to add control lines to perform the first and second imaging for the first pixels, but since an entire row is made up of first pixels, the number of control lines relative to the number of first pixels can be minimized.

As described above, the image sensor according to this embodiment performs the first readout process and the first signal processing only in the first pixel group, thereby reducing the load of signal processing and transfer processing and enabling efficient event detection.

Note that the pixel arrangement in FIG. 7A is only one example. For example, as shown in FIG. 7B, the pixels of the first pixel group 601 can be arranged in a dot pattern. This increases the number of pixel control lines, but the number of pixel data read out in the first readout process can be further reduced. Therefore, an imaging element can be configured that further reduces power consumption and shortens the time required for event detection. FIG. 7B shows a configuration in which the first pixels are arranged separately one by one in a dot pattern, but a configuration in which multiple first pixels are arranged together in a tile pattern may also be adopted. In addition, the first pixels do not need to be arranged in a lattice pattern, and may be arranged in any position. FIG. 7C shows a configuration in which the first pixels are arranged in a tile pattern and at non-periodic intervals. The pixel arrangement that can be adopted in this embodiment is not limited to the above examples and may be any.

(Fourth embodiment)
The fourth embodiment is different from the third embodiment in the second readout process for the second pixel group, but is otherwise the same as the third embodiment. The following mainly describes the differences from the third embodiment.

The image sensor according to this embodiment includes a first pixel group and a second pixel group, similar to the third embodiment. In this embodiment, during a period in which the pixels included in the first pixel group are performing the first imaging, the pixels included in the second pixel group start exposure for the second imaging.

The operation timing of the second pixel group will be explained using FIG. 8. Note that the operation timing of the pixels belonging to the first pixel group is the same as that explained using FIG. 3, so the explanation will be omitted.

The operation of the second pixel group (FIG. 8) differs from the operation of the first pixel group (FIG. 3) in that there is no first exposure period, first readout, or first signal processing for the second pixel group, as shown in (b) to (d) of FIG. 8. In this embodiment, the start timing of the first exposure of the first pixel group and the start timing of the second exposure of the second pixel group are the same, and the exposure period of the first exposure of the first pixel group and the exposure period of the second exposure of the second pixel group overlap in time. Note that the effect of this embodiment can be obtained even if the exposure start timings are not coincident, by overlapping the exposure periods and lengthening the exposure period of the second exposure of the second pixel group as in the third embodiment.

In this embodiment, the pixels belonging to the second pixel group can use the long period from the reset signal for the accumulated charge to the second readout signal as an exposure period, as shown in FIG. 8(e). This means that for the second pixel group, even if an event is detected, there is no need to shorten the exposure time during image capture, and the image quality of the frame image of the pixels in the second pixel group is not affected by event detection.

The imaging element of this embodiment can detect events in real time while minimizing degradation of the image quality of the captured image.

Fifth Embodiment
The fifth embodiment is different from the fourth embodiment in the content of the frame image generation process, but is otherwise the same as the fourth embodiment. The following mainly describes the differences from the fourth embodiment.

In this embodiment, the pixel signal obtained from the first pixel during the second imaging and the pixel signal obtained from the second pixel during the second imaging are synthesized (corrected) based on the ratio of the lengths of the exposure periods to generate pixel output data.

This embodiment will be described with reference to FIG. 9A. In this embodiment, as in the fourth embodiment, during the period when the pixels included in the first pixel group are performing the first imaging, the pixels included in the second pixel group start exposure in the second imaging. Therefore, as shown in FIG. 9A, the length of the exposure period T1 during the second imaging of the pixels included in the first pixel group is different from the length of the exposure period T2 during the second imaging of the pixels included in the second pixel group. Specifically, the pixels included in the first pixel group perform the first imaging and the first signal processing additionally compared to the pixels included in the first pixel group, so the exposure time during the second imaging is shorter. Therefore, if the acquired pixel signal is converted directly into a frame image, the pixels included in the first image group will be darker than the pixels included in the second pixel group.

In consideration of this problem, the signal processing unit 140 in this embodiment performs the following processing as the second signal processing. The signal processing unit 140 first adjusts each pixel value of the second pixel signal of the first pixel group by multiplying it by the ratio between the second exposure time of the second pixel group and the first pixel group. The signal processing unit 140 then generates an image signal for output based on the adjusted second pixel signal of the first pixel group and the second pixel signal of the second pixel group.

9B is a diagram for explaining the frame image generation process by the signal processing unit 140 in this embodiment. In this embodiment, the signal processing unit 140 has a correction unit 141 that multiplies the second pixel signal 221 by the second imaging of the first pixel with a predetermined gain to increase the pixel value. The predetermined gain is the ratio (T2/T1) of the time length T1 information 211 of the exposure period at the second imaging of the first pixel to the time length T2 information 212 of the exposure period at the second imaging of the second pixel. By multiplying the pixel signal 221 of the first pixel by the gain T2/T1, the corrected pixel signal 231 has the same brightness as the pixel signal 222. The signal processing unit 140 combines the corrected pixel signal 231 by the correction unit 141 with the pixel signal 222 by the second imaging of the second pixel to output the frame image 203.

In this way, the brightness of the first pixel and the second pixel can be made almost uniform, and the change in brightness of the frame image can be reduced to a minimum. Note that, although noise also increases by applying a gain to the pixel signal of the first pixel, the number of first pixels can be made smaller than the number of second pixels, so that the degradation of image quality due to the correction of the pixel signal of the first pixel can be kept to a minimum.

As described above, the imaging element of this embodiment can perform immediate event detection by minimizing changes in brightness of the image quality of the captured image.

Sixth Embodiment
The sixth embodiment differs from the fourth embodiment in that a pixel signal in the first imaging of the first pixel used for event detection is also used for generating a frame image, but is otherwise the same as the fourth embodiment. Note that this embodiment may be implemented in combination with the fifth embodiment.

FIG. 10 is a schematic diagram illustrating the signal processing unit 140 of this embodiment. The signal processing unit 140 generates a frame image 203 using the first pixel signal 220 and the second pixel signal 221 of the first pixel group, and the second pixel signal 222 of the second pixel group. The first pixel signal 220 obtained in the first imaging of the first pixel is used for event detection and then temporarily stored in the internal memory of the signal processing unit 140. The first pixel signal 220 is then summed by the adder unit 142 with the second pixel signal 221 of the first pixel at the time of the second imaging. The signal after the summation is referred to as a pixel signal 232. The signal processing unit 140 adds the pixel signal 232 to the pixel signal 222 of the second pixel at the time of the second imaging to generate the frame image 203.

According to this embodiment, the exposure time of the pixel signal used to generate a frame image for the first pixel is the sum of the first exposure time T0 (FIG. 3(b)) and the second exposure time T1 (FIG. 3(e)). The sum of the exposure time is approximately the same as the exposure period T2 (FIG. 8(e)) during the second shooting of the second pixel. Therefore, the difference in brightness between the first pixel and the second pixel can be suppressed, and a high-quality frame image can be generated. In addition, since the magnification of the pixel signal is not changed as in the fifth embodiment, there is no deterioration in the signal-to-noise ratio (SNR), and a better quality image can be output.

Therefore, the imaging element according to this embodiment can perform immediate event detection while minimizing degradation of the image quality of the captured image.

In the description of this embodiment, the pixel signal 232 and the pixel signal 222 are simply added together, but the present invention is not limited to this. If there is no problem in practical use, the above configuration may be used, or if a pixel with higher image quality is required, a configuration combined with the configuration of the fifth embodiment may be adopted. Specifically, the signal processing unit 140 may increase the pixel signal 232 obtained by adding the pixel signal 220 and the pixel signal 221 by the ratio of the sum (T0+T1) of the first exposure time and the second exposure time of the first pixel to the second exposure time (T2) of the second pixel. This makes it possible to correct the difference in exposure time for the first readout time for event detection, and to output a frame image with higher image quality.

Seventh Embodiment
The seventh embodiment differs from the first embodiment in that the first imaging (exposure and readout) is repeated multiple times within one period, and is otherwise the same as the first embodiment. Note that this embodiment may be implemented in combination with any of the second to sixth embodiments.

Figure 11 is a diagram illustrating the operation of the image sensor in this embodiment. In this embodiment, as shown in Figure 11, the first exposure and first readout process are repeated multiple times during the first imaging period in one cycle. In Figure 11, three exposures and readouts are performed during the first imaging period, but the number of times is not particularly limited as long as it is two or more. By performing multiple exposures and readouts during the first imaging period, event detection processing (first signal processing) can be performed based on multiple first pixel signals, making it possible to more accurately grasp the moving state and moving direction of the subject. Specifically, the signal processing unit 140 can grasp high-speed movement within one frame, and more accurately detect events.

The imaging element of this embodiment can provide an imaging element that can perform event detection more accurately and capture images without delay when an event occurs, without performing external signal processing or image transfer that places a high load on the user.

Eighth embodiment
The eighth embodiment is different from the first to seventh embodiments in that the imaging element is configured as a stacked sensor, but is otherwise the same as the first to seventh embodiments. The imaging element according to this embodiment is configured as a stacked type sensor in which a substrate having a photoelectric conversion element (first substrate) and a digital substrate (second substrate) are stacked in the substrate thickness direction.

Figure 12 is a diagram illustrating the configuration of the imaging element according to this embodiment. In the imaging element according to this embodiment, a substrate 651 having a photoelectric conversion unit 110 and a digital substrate 652 are stacked in the substrate thickness direction and connected by a substrate joint member 653. The substrate 651 having the photoelectric conversion element is equipped with circuits close to the pixels, such as the photoelectric conversion unit 110, the signal readout unit 130, the pixel selection unit 132, and the signal transfer unit 133. On the other hand, the digital substrate 652 is equipped with a digital signal conversion unit 134, a signal processing unit 140, a memory, and the like. By connecting these two substrates in the vertical direction, signals from each pixel can be efficiently transferred to the digital unit, resulting in a smaller, higher performance imaging element. In addition, the substrate joint member 653 can be easily realized by using a general semiconductor manufacturing process, such as solder bumps or direct bonding between substrates.

In this embodiment, a stacked type sensor is used, which allows high-speed signal processing of pixel signals from a large number of pixels, resulting in more accurate and faster event detection.

Ninth embodiment
The ninth embodiment is an imaging device using any one of the imaging elements described in the first to eighth embodiments.

Figure 13A is a schematic diagram illustrating an imaging device 701 of this embodiment. The imaging device 701 includes an imaging element 100 according to any one of the first to eighth embodiments, and a recording device 702. The imaging element 100 outputs an image output 204 to the recording device 702. The image output 204 is a valid frame image when an event has occurred, and is an invalid frame image when no event has occurred. The recording device 702 records a valid frame image when it is input, and does not record when an invalid frame image is input.

In this way, by configuring the imaging device 701 using the imaging element 100, it is possible to monitor a subject event and automatically record the moment the event occurs.

In the above description, a form in which only the moment an event occurs has been described, but it can also be used in a similar manner in which it records for a certain period of time after the event occurs. Also, instead of or in addition to recording an image at the time of event detection in the recording device 702, the imaging device 701 may have a function of transmitting the image to the outside via a network, etc.

Tenth Embodiment
The tenth embodiment is a monitoring device using any one of the imaging elements described in the first to eighth embodiments.

FIG. 13B is a schematic diagram for explaining a monitoring device 800 of this embodiment. The monitoring device 800 includes the image sensor 100 according to any one of the first to eighth embodiments and an event notification unit 802. The image sensor 100 outputs an image output 204 to the event notification unit 802. The image output 204 is a valid frame image when an event occurs, and is an invalid frame image when an event does not occur. When a valid frame image is input, the event notification unit 802 notifies that an event has occurred, and does not notify when an invalid frame image is input. The specific manner of notification is not particularly limited. For example, the event notification unit 802 may notify an external device that an event has occurred by communication, or may notify surrounding people that an event has occurred by generating sound or light.

For example, the monitoring device 800 can be installed to capture images of locations such as intersections, and can be configured to detect the occurrence of a traffic accident as an event. When a traffic accident is detected, the monitoring device 800 can be configured to automatically notify the police and report the situation to a traffic information center. The monitoring device 800 can also directly broadcast images to nearby vehicles. This allows the driver to assess the situation on the scene for themselves based on the images, and take action to avoid danger or traffic jams, etc.

The monitoring device 800 can also be used as a danger prevention system at intersections with poor visibility. The monitoring device 800 is configured to detect the approach of a car or pedestrian as an event, and reports the approach to those around it to issue a warning.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

100: Image sensor 110: Photoelectric conversion section 120: Reset section 130: Signal readout section 140: Signal processing section

Claims (22)

An imaging element having a plurality of pixels,
The accumulated charges of the plurality of pixels are periodically reset;
In at least a portion of the plurality of pixels,
performing a first readout process for reading out accumulated charges due to the first exposure;
performing a first signal process to detect whether or not an event has occurred based on a first pixel signal read out by the first readout process;
a determination is made according to a result of the first signal processing as to whether or not to perform a second readout process for reading out accumulated charges due to a second exposure and a second signal processing for correcting a second pixel signal read out by the second readout process so that the second pixel signal is suitable as an image ;
the first readout process, the first signal processing, and the second readout process are performed within one period of a reset that is performed periodically;
Image sensor. the first readout process, the first signal process, the second readout process, and the second signal process are performed within one period of a reset that is performed periodically;
The imaging device according to claim 1 . the second signal processing is processing for generating an image signal for output based on the first pixel signal in addition to the second pixel signal;
The imaging device according to claim 1 . the plurality of pixels includes a first group of pixels and a second group of pixels;
the first readout process is performed on pixels belonging to the first pixel group;
the second readout process is performed on pixels belonging to the first pixel group and pixels belonging to the second pixel group;
The imaging device according to claim 3 . an exposure period of the first exposure for the pixels belonging to the first pixel group and an exposure period of the second exposure for the pixels belonging to the second pixel group overlap in time;
The imaging device according to claim 4 . a start timing of the first exposure for the pixels belonging to the first pixel group is equal to a start timing of the second exposure for the pixels belonging to the second pixel group;
The imaging device according to claim 5 . the second signal processing is processing for generating an image signal for output based on the second pixel signal;
The imaging device according to claim 4 . the second signal processing is a process of adjusting each pixel value of a second pixel signal of a pixel belonging to a first pixel group by a ratio between an exposure time of a second exposure of the pixel belonging to the second pixel group and an exposure time of a second exposure of the pixel belonging to the first pixel group, and generating the image signal for output using the adjusted second pixel signal.
The imaging device according to claim 7. the second signal processing is processing for generating the image signal for output using a first pixel signal and a second pixel signal of a pixel belonging to a first pixel group and a second pixel signal of a pixel belonging to a second pixel group;
The imaging device according to claim 8 . the second signal processing is a process of adjusting each pixel value, which is a sum of a first pixel signal and a second pixel signal of a pixel belonging to a first pixel group, by a ratio between an exposure time of a second exposure of the pixels belonging to the second pixel group and a total exposure time of the first exposure and the second exposure of the pixels belonging to the first pixel group, and generating the image signal for output using the adjusted image signal;
The imaging device according to claim 9 . the number of pixels belonging to the first pixel group is smaller than the number of pixels belonging to the second pixel group;
The imaging device according to claim 4 . The first exposure and the first readout process are performed a plurality of times within one reset cycle.
The imaging device according to claim 1 . the first signal processing is processing for detecting an occurrence of a predetermined event,
When the occurrence of the predetermined event is detected, the second readout process and the second signal process are performed.
The imaging device according to claim 1 . when occurrence of the predetermined event is detected, the second readout process and the second signal process are performed continuously for a predetermined period of time.
The imaging device according to claim 13. the predetermined event includes at least one of a still subject starting to move and a moving subject becoming still;
The imaging device according to claim 14. when the occurrence of the predetermined event is not detected in the first signal processing, the second readout processing and the second signal processing are not performed, and an invalid image signal is output.
The imaging device according to claim 13 . The invalid image signal is a signal representing an image filled with a specific value or having a specific pattern.
The imaging device according to claim 16. a first substrate having a photoelectric conversion unit and a readout unit that reads out charges stored in the photoelectric conversion unit;
a second substrate having a signal processing unit that performs the first signal processing and the second signal processing;
Equipped with
The first substrate and the second substrate are laminated in a substrate thickness direction.
The imaging device according to claim 1 . The imaging device according to claim 1 ,
A recording device;
Equipped with
the first signal processing is processing for detecting an occurrence of a predetermined event,
the recording device records the image signal generated by the second signal processing when the occurrence of the predetermined event is detected.
Imaging device. The imaging device according to claim 1 ,
A notification unit;
Equipped with
the first signal processing is processing for detecting an occurrence of a predetermined event,
The notification unit notifies the occurrence of the predetermined event when the occurrence of the predetermined event is detected.
monitoring equipment. the notification unit notifies the occurrence of the predetermined event, and also notifies the occurrence of the image signal generated by the second signal processing.
21. A monitoring device according to claim 20. A method for controlling an image sensor including a plurality of pixels each including a photoelectric conversion unit, a readout unit that reads out charges accumulated in the photoelectric conversion unit, and a reset unit that resets the photoelectric conversion unit, and a signal processing unit that performs signal processing on pixel signals output from the plurality of pixels, comprising:
periodically resetting the photoelectric conversion units in the plurality of pixels;
performing a first readout process for reading out accumulated charges due to a first exposure;
performing a first signal processing step of detecting whether or not an event has occurred based on a first pixel signal read out by the first readout processing;
a step of determining whether or not to perform a second readout process for reading out accumulated charges due to a second exposure and a second signal process for correcting a second pixel signal read out by the second readout process so that the second pixel signal is suitable as an image , according to a result of the first signal processing;
Including,
the first readout process, the first signal processing, and the second readout process are performed within one period of a reset that is performed periodically;
A method for controlling an image sensor.

Images