JP7669404B2 - IMAGING APPARATUS, PROCESSING METHOD AND PROGRAM FOR IMAGING APPARATUS - Google Patents

Description

This disclosure relates to an imaging device, a processing method for the imaging device, and a program.

In recent years, photoelectric conversion devices have become known that digitally count the number of photons arriving at an avalanche photodiode (APD) and output the count value from a pixel as a photoelectrically converted digital signal. Non-Patent Document 1 discloses that a phenomenon called avalanche emission occurs in a photoelectric conversion device having an APD.

I. Rech et al. , “Optical crosstalk in single photon avalanche diode arrays: a new complete model”, OpEx 16(12), 2008

In photoelectric conversion devices with APDs, there is a problem that the DCR changes as the usage time increases, causing the output signal to change. DCR is the dark count rate, which is the average count value when there is no incident light.

An object of the present disclosure is to make it possible to suppress the effects of deterioration of the imaging unit.

The imaging device includes an imaging unit including an avalanche photodiode, an estimate calculation unit that calculates an estimate value indicating a degree of deterioration of the imaging unit based on at least one of information regarding the number of avalanche emissions in the avalanche photodiode, exposure conditions of the imaging unit, and a temperature of the imaging unit, and an image processing unit that performs image processing on an output signal of the imaging unit based on the estimate value indicating the degree of deterioration and an image processing parameter based on a first threshold value.

According to the present disclosure, the influence of deterioration of the imaging section can be suppressed.

FIG. 2 is a diagram illustrating a configuration example of a photoelectric conversion element. FIG. 4 is a diagram illustrating a configuration example of a sensor substrate. FIG. 2 is a diagram illustrating an example of the configuration of a circuit board. 2 is a diagram of an equivalent circuit of a signal processing circuit corresponding to a pixel of a photoelectric conversion element. 4 is a timing chart of a photoelectric conversion element. FIG. 1 is a diagram illustrating an example of the configuration of an imaging system. FIG. 1 is a diagram illustrating an example of the configuration of a surveillance camera. 10 is a flowchart showing a processing method of the surveillance camera. FIG. 1 is a diagram illustrating an example of the configuration of a surveillance camera. 10 is a flowchart showing a processing method of the surveillance camera.

The present embodiment will be described in detail below. Note that the embodiment described below is merely an example, and should be modified or adjusted as appropriate depending on the configuration of the device to which the present embodiment is applied and various conditions, and is not limited to the following embodiment. Also, in all figures, parts having the same function are given the same numbers, and repeated explanations will be omitted.

First Embodiment
<Introduction>
1 is a diagram showing a configuration example of a photoelectric conversion element 100 according to the first embodiment. In the following, a photoelectric conversion element having a so-called laminated structure in which the photoelectric conversion element 100 is configured by laminating two substrates, a sensor substrate 11 and a circuit substrate 21, and electrically connecting the sensor substrate 11 and the circuit substrate 21, will be described as an example. However, the photoelectric conversion element 100 may have a so-called non-laminated structure in which the configuration included in the sensor substrate 11 and the configuration included in the circuit substrate 21 are arranged on a common semiconductor layer. The sensor substrate 11 has a pixel region 12. The circuit substrate 21 has a circuit region 22 that processes a signal generated in the pixel region 12.

<Sensor board>
Fig. 2 is a diagram showing an example of the configuration of the sensor substrate 11 in Fig. 1. The pixel region 12 of the sensor substrate 11 includes a plurality of pixels 101 arranged two-dimensionally across a plurality of rows and a plurality of columns. Each of the plurality of pixels 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD). The photoelectric conversion unit 102 converts light into an electric charge by photoelectric conversion. Note that the number of rows and columns of the pixels 101 is not limited.

<Circuit board>
Fig. 3 is a diagram showing an example of the configuration of the circuit board 21 in Fig. 1. The circuit board 21 has a signal processing circuit 103 that processes charges photoelectrically converted by the photoelectric conversion unit 102 in Fig. 2, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a signal line 113, a vertical scanning circuit 110, and an output circuit 114.

The control pulse generation unit 115 supplies control pulses to the vertical scanning circuit 110, the horizontal scanning circuit 111, and the readout circuit 112.

The vertical scanning circuit 110 receives a control pulse supplied from the control pulse generating unit 115 and supplies a selection pulse for each row of the multiple signal processing circuits 103. The vertical scanning circuit 110 uses logic circuits such as a shift register and an address decoder.

The signal processing circuit 103 processes the signal output from the photoelectric conversion unit 102 of the pixel 101 in FIG. 2. The signal processing circuit 103 has a counter, a memory, etc., and digital values are stored in the memory. The signal processing circuit 103 outputs digital pixel signals based on the signals output from the photoelectric conversion unit 102 to the signal line 113 of each column on a row-by-row basis in response to a selection pulse.

The readout circuit 112 receives a control pulse supplied from the control pulse generation unit 115 and holds the pixel signals of the signal lines 113 of each column.

The horizontal scanning circuit 111 receives a control pulse supplied from the control pulse generating unit 115 and outputs a control pulse that sequentially selects each column to the readout circuit 112. The readout circuit 112 sequentially outputs the pixel signals of each column to the output circuit 114 in response to the control pulse. The output circuit 114 sequentially outputs the pixel signals to the outside of the photoelectric conversion element 100.

<Connection between sensor board and circuit board>
2 and 3, a plurality of signal processing circuits 103 are arranged in each of the regions overlapping the plurality of pixels 101 in a plan view. A vertical scanning circuit 110, a horizontal scanning circuit 111, a readout circuit 112, an output circuit 114, and a control pulse generating unit 115 are arranged so as to overlap a region between an end of the sensor substrate 11 and an end of the pixel region 12 in a plan view. In other words, the sensor substrate 11 has the pixel region 12 and a non-pixel region arranged around the pixel region 12. The vertical scanning circuit 110, the horizontal scanning circuit 111, the readout circuit 112, the output circuit 114, and the control pulse generating unit 115 are arranged in a region overlapping the non-pixel region in a plan view.

The arrangement of the signal lines 113, readout circuits 112, and output circuits 114 is not limited to that shown in FIG. 3. For example, the signal lines 113 may be arranged to extend in the row direction, and the readout circuits 112 may be arranged at the end of the signal lines 113. Also, it is not necessary to provide one signal processing circuit 103 for each photoelectric conversion unit 102. One signal processing circuit 103 may be configured to share multiple photoelectric conversion units 102 and perform signal processing sequentially.

<Pixel equivalent circuit>
FIG. 4 is an equivalent circuit diagram showing a configuration example of the photoelectric conversion unit 102 and the signal processing circuit 103 corresponding to the photoelectric conversion unit 102 in FIGS.

The photoelectric conversion unit 102 has an avalanche photodiode (APD) 201. The APD 201 generates a pair of charges according to incident light by photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VL is supplied. The other of the two nodes of the APD 201 is connected to a power supply line to which a driving voltage VH higher than the driving voltage VL is supplied via a quench element 202. In FIG. 4, one node of the APD 201 is an anode, and the other node of the APD 201 is a cathode. A reverse bias voltage VR is supplied to the anode and cathode of the APD 201 so that the APD 201 performs an avalanche multiplication operation. Here, the voltage VR is calculated by the following formula (1).
VR=VL-VH (1)

When this voltage VR is supplied to the APD 201, the charge generated by the incident light undergoes avalanche multiplication, generating an avalanche current.

When voltage VR is supplied, there are Geiger mode and linear mode. Geiger mode is a mode in which APD201 operates with a voltage difference between the anode and cathode that is greater than the breakdown voltage. Linear mode is a mode in which APD201 operates with a voltage difference between the anode and cathode that is close to or less than the breakdown voltage. APD201 operated in Geiger mode is called a single photon avalanche diode (SPAD). In the case of a SPAD, for example, the drive voltage VL is -30V and the drive voltage VH is 1V.

The quench element 202 is connected to a power supply line through which the drive voltage VH is supplied and to one of the anode and cathode nodes of the APD 201. The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication of the APD 201, and suppresses the voltage supplied to the APD 201 to suppress avalanche multiplication (quench operation). The quench element 202 also has the function of returning the voltage supplied to the APD 201 to the drive voltage VH by passing a current equivalent to the voltage drop caused by the quench operation (recharge operation).

The signal processing circuit 103 has a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a selection circuit 212. In FIG. 4, the signal processing circuit 103 has the quench element 202, the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212, but is not limited to this. The signal processing circuit 103 may have at least one of the waveform shaping unit 210, the counter circuit 211, and the selection circuit 212.

The waveform shaping unit 210 shapes the waveform of the signal (voltage) of the cathode of the APD 201 obtained when photons are detected, and outputs a pulse signal. For example, an inverter circuit is used as the waveform shaping unit 210. In FIG. 4, an example in which one inverter is used as the waveform shaping unit 210 is shown, but a circuit in which multiple inverters are connected in series may be used, or other circuits that have a waveform shaping effect may be used.

The counter circuit 211 counts the pulse signal output from the waveform shaping unit 210 and holds the count value. In addition, when a control pulse RES is supplied via the drive line 213, the counter circuit 211 resets the signal held in the counter circuit 211.

The selection circuit 212 receives a control pulse SEL from the vertical scanning circuit 110 in FIG. 3 via the drive line 214, and switches between electrical connection and disconnection between the counter circuit 211 and the signal line 113. The selection circuit 212 includes, for example, a buffer circuit for outputting a signal, and outputs the count value from the counter circuit 211 to the signal line 113. In FIG. 4, an example of switching using the selection circuit 212 is shown, but the electrical connection may be switched by arranging a switch such as a transistor between the quench element 202 and the APD 201, or between the photoelectric conversion unit 102 and the signal processing circuit 103. Similarly, the supply of the drive voltage VH or drive voltage VL supplied to the photoelectric conversion unit 102 may be electrically switched using a switch such as a transistor.

<Circuit Driving>
5 is a diagram showing the relationship between the operation of the APD 201 in FIG. 4 and the output signal. The input side of the waveform shaping unit 210 in FIG. 4 is nodeA, and the output side of the waveform shaping unit 210 is nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to the APD 201. When a photon is incident on the APD 201 at time t1, avalanche multiplication occurs in the APD 201, an avalanche multiplication current flows through the quench element 202, and the voltage of nodeA drops. When the voltage drop amount of nodeA becomes larger and the potential difference applied to the APD 201 becomes smaller, as at time t2, the avalanche multiplication of the APD 201 stops, and the voltage level of nodeA does not drop by more than a certain value. Then, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from the drive voltage VL, and at time t3, nodeA is stabilized to its original potential level. At this time, the waveform shaping section 210 outputs a signal for nodeB that sets the portion of the waveform value of nodeA that is lower than the waveform threshold to a high level and sets the portion of the waveform value of nodeA that is higher than the waveform threshold to a low level.

Figure 6 is a diagram showing an example of the configuration of an imaging system 600 according to this embodiment. The imaging system 600 has a surveillance camera 601, a network 602, a client device 603, an input device 604, and a display device 605. The surveillance camera 601 is an example of an imaging device, has the photoelectric conversion element 100 of Figure 1, captures moving images, and processes the moving images. The surveillance camera 601 and the client device 603 can communicate with each other via the network 602. The input device 604 and the display device 605 are connected to the client device 603. The client device 603 is an example of an information processing device.

Figure 7 is a diagram showing an example of the configuration of the surveillance camera 601 in Figure 6. The surveillance camera 601 has a photoelectric conversion element 100, an optical system 701, a CPU 702, a ROM 703, a RAM 704, an imaging system control unit 705, and a control unit 706. The surveillance camera 601 further has an estimated value calculation unit 707, a camera setting acquisition unit 708, a threshold calculation unit 709, a parameter setting unit 710, an image processing execution unit 711, an encoder unit 712, and a network I/F 713.

The optical system 701 has a zoom lens, a focus lens, a blur correction lens, an aperture, and a shutter, and collects optical information from the subject. The photoelectric conversion element 100 is an example of an imaging unit and has the APD 201 described in Figures 1 to 5, and acquires color information by combining with a color filter or the like. As the usage time of the photoelectric conversion element 100 increases, the DCR changes and the output count value changes. The DCR is the dark count rate, and is the average value of the count value of the counter circuit 211 (Figure 4) detected when there is no incident light.

The CPU 702 is involved in all processing of each component, sequentially reading and interpreting programs stored in the ROM (Read Only Memory) 703 and the RAM (Random Access Memory) 704, and executing processing according to the results.

The imaging system control unit 705 performs control of the optical system 701 in accordance with instructions from the CPU 702, such as adjusting the focus, opening the shutter, and adjusting the aperture. The control unit 706 performs control in accordance with instructions from the client device 603.

The estimated value calculation unit 707 quantitatively calculates an estimated value of the degree of change in the aging of the photoelectric conversion element 100 having the APD 201. Here, the estimated value of the degree of change is calculated based on information such as the number of avalanche emissions since the start of operation, the exposure conditions (aperture, shutter, gain), and temperature.

The camera setting acquisition unit 708 acquires setting information of various camera functions as camera setting information from the imaging system control unit 705 and the control unit 706. The threshold calculation unit 709 calculates a change threshold for determining the degree of change based on the camera setting information obtained from the camera setting acquisition unit 708. Here, the above-mentioned camera setting information refers to, for example, the codec setting of the compression process of the downstream encoder unit 712, and setting information such as tone correction, noise reduction processing, defective pixel correction, and demosaicing processing included in the image processing execution unit 711.

The parameter setting unit 710 compares the change degree estimate calculated by the estimate calculation unit 707 with the change threshold calculated by the threshold calculation unit 709 to calculate and set image processing parameters. Here, the image processing parameters refer to parameters such as tone correction processing, noise reduction processing, defective pixel correction processing, saturation enhancement processing, and demosaicing processing executed by the image processing execution unit 711.

The image processing execution unit 711 is an image processing unit that performs image processing on the count value output by the photoelectric conversion element 100 based on the image processing parameters set by the parameter setting unit 710, and outputs image data. Examples of image processing include tone correction processing, noise reduction processing, defective pixel correction processing, saturation enhancement processing, and demosaicing processing.

The encoder unit 712 converts the image data output from the image processing execution unit 711 into a file format such as Motion JPEG, H264, or H265. The network I/F 713 is an interface used for communication with external devices such as the client device 603 via the network 602.

The network 602 is a network that connects the surveillance camera 601 and the client device 603. The network is composed of, for example, multiple routers, switches, cables, etc. that satisfy a communication standard such as Ethernet (registered trademark). In this embodiment, the network 602 may be of any communication standard, scale, or configuration as long as it can perform communication between the surveillance camera 601 and the client device 603. For example, the network 602 may be composed of the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc.

Figure 8 is a flowchart showing a processing method of the surveillance camera 601 in Figure 7. The processing of the flowchart in Figure 8 may be executed by hardware or software, or may be partially realized by a software configuration and the rest by a hardware configuration. When executed by software, the processing of the flowchart in Figure 8 is realized by the CPU 702 executing a program stored in, for example, the ROM 703. The same applies to the flowchart in Figure 10 described later.

In step S801, the estimate calculation unit 707 estimates the degree of change due to aging of the photoelectric conversion element 100 having the APD 201, and calculates an estimate of the degree of change of the photoelectric conversion element 100. For example, the DCR of the photoelectric conversion element 100 changes as the usage time increases, and the count value output by the photoelectric conversion element 100 changes. The estimate calculation unit 707 calculates the degree of change estimate based on the number of avalanche emissions since the start of operation, the exposure conditions, the temperature, or the brightness information of the subject. As an example, the estimate calculation unit 707 calculates the cumulative value of brightness since the start of operation based on the count value output by the photoelectric conversion element 100, and calculates the degree of change estimate based on the cumulative value of brightness. The larger the cumulative value of brightness, the larger the degree of change estimate. The larger the degree of change estimate, the larger the degree of change, and the smaller the degree of change estimate, the smaller the degree of change. The degree of change includes the degree of deterioration.

Next, in step S802, the camera setting acquisition unit 708 acquires setting information of various camera functions from the imaging system control unit 705 and the control unit 706 as camera setting information (distribution setting information). The camera setting information is setting information of the surveillance camera 601. In this embodiment, an embodiment in which codec setting information of the compression process obtained from the downstream encoder unit 712 is used as an example of camera setting information will be described. Specifically, the camera setting acquisition unit 708 acquires codec setting information of the encoding method, bit rate, and Qp (Quantization Parameter) value of the encoder unit 712 as the camera setting information.

The following describes the processing and issues related to the codec setting information acquired in this embodiment. In general, in an imaging system 600 that distributes video via a network 602 as shown in FIG. 6, there are restrictions on the capacity and distribution bandwidth of the network server, and therefore video distribution with a reduced bit rate is required. As such, there are methods known as contrast bitrate (CBR) encoding and variable bitrate (VBR) encoding as encoding methods for compressing video. CBR encoding is a method of encoding data so that it always has a constant bit rate, and VBR encoding is a method of encoding data by varying the bit rate value.

With CBR encoding, image quality changes depending on the bit rate that is set. When a low bit rate is set, frequencies over a wide range, including not only high-frequency components but also low-frequency components, are compressed, resulting in discontinuous changes in brightness, color tone, and edges between blocks, creating an issue known as block noise.

It is known that similar issues can occur with VBR encoding, depending on the Qp value setting. In general, the smaller the Qp value, the higher the image quality and the larger the data size, while the larger the value, the lower the image quality and the more likely block noise is to occur. Images with block noise undergo strong compression processing, including high-frequency noise and defective pixel noise components.

The above is the process and issues related to the codec setting information acquired in this embodiment, and in steps S803 and onwards, which will be described later, a change threshold is set that takes into account the issue of block noise.

Next, in steps S803 to S809, the threshold calculation unit 709 calculates a change threshold for determining the degree of change based on the camera setting information described above. First, in step S803, the threshold calculation unit 709 determines whether the encoding method is VBR encoding or CBR encoding based on the encoding method information in the camera setting information. If it is CBR encoding, processing proceeds to step S804, and if it is VBR encoding, processing proceeds to step S807.

In step S804, the threshold calculation unit 709 determines whether the bit rate of the CBR encoding is equal to or greater than the bit rate threshold based on the bit rate information in the camera setting information. The bit rate threshold is, for example, 8 Mbps. If the bit rate is equal to or greater than the bit rate threshold, the process proceeds to step S805, and if the bit rate is not equal to or greater than the bit rate threshold, the process proceeds to step S806.

In step S805, the threshold calculation unit 709 sets a small first value as the change threshold. The threshold calculation unit 709 is an example of a threshold setting unit. Then, the process proceeds to step S810.

In step S806, the threshold calculation unit 709 sets a second value greater than the first value as the change threshold. Processing then proceeds to step S810.

In step S807, the threshold calculation unit 709 determines whether the Qp value of the VBR encoding is equal to or less than the Qp threshold based on the Qp value information in the camera setting information. The Qp threshold is, for example, 10. If the Qp value is equal to or less than the Qp threshold, processing proceeds to step S808, and if the Qp value is not equal to or less than the Qp threshold, processing proceeds to step S809.

In step S808, the threshold calculation unit 709 sets a small third value as the change threshold. Then, the process proceeds to step S810.

In step S809, the threshold calculation unit 709 sets a fourth value, which is greater than the third value, as the change threshold. Then, the process proceeds to step S810.

As described above, the change threshold set in steps S803 to S809 is used in comparison with the change degree estimated value in steps S810 to S812. When the change threshold is small, the bit rate is set high (or the Qp value is small), so there is a high possibility that the effects of DCR noise (dark current noise) and defective pixel noise due to aging will become apparent due to the compression process in the subsequent stage. On the other hand, when the change threshold is large, the bit rate is set low (or the Qp value is large), so there is a high possibility that the effects of DCR noise and defective pixel noise due to aging will be small due to the strong compression process in the subsequent stage. Taking into account the relationship between the change threshold and image quality as described above, in steps S811 or S812, the parameter setting unit 710 sets the optimal image processing parameters.

Next, in steps S810 to S812, the parameter setting unit 710 determines whether the change degree estimated value calculated by the estimate calculation unit 707 is equal to or greater than the change threshold value set by the threshold calculation unit 709. If the change degree estimated value is equal to or greater than the change threshold value, the process proceeds to step S811, and if the change degree estimated value is not equal to or greater than the change threshold value, the process proceeds to step S812.

In step S811, the parameter setting unit 710 sets the first image processing parameters for strongly reducing noise and defective pixel noise. Then, the process proceeds to step S813.

In step S812, the parameter setting unit 710 sets, with respect to the image processing parameters, second image processing parameters for leaving noise or defective pixel noise as is. The first image processing parameters in step S811 are parameters for performing processing that reduces noise or defective pixel noise more strongly than the second image processing parameters in step S812. Processing then proceeds to step S813.

Finally, in step S813, the image processing execution unit 711 performs image processing on the count value (output signal) output by the photoelectric conversion element 100 based on the first image processing parameter or the second image processing parameter set by the parameter setting unit 710.

As described above, the surveillance camera 601 sets the change threshold value taking into account the effect of the compression process of the encoder unit 712, and sets optimal image processing parameters according to the change degree estimate value and the change threshold value. Therefore, the surveillance camera 601 can optimally set the parameters of the image processing execution unit 711 located at the front stage and the encoder unit 712 located at the rear stage in response to the aging of the photoelectric conversion element 100 having the APD 201 (increase in DCR noise and defective pixel noise). If the compression process of the encoder unit 712 is not taken into account, problems arise such as unnecessary over-correction of noise and defective pixel noise, or insufficient correction, but this embodiment makes it possible to solve these problems. The surveillance camera 601 of this embodiment can suppress the effects of aging of the photoelectric conversion element 100 caused by increased use time.

Second Embodiment
In the second embodiment, a configuration will be described in which image processing parameters are set for each region based on an estimated value of the degree of change for each region.

Figure 9 is a diagram showing an example of the configuration of a surveillance camera 601 according to the second embodiment. The surveillance camera 601 in Figure 9 is different from the surveillance camera 601 in Figure 7 in that instead of the estimate value calculation unit 707, parameter setting unit 710, and image processing execution unit 711, a region-specific estimate value calculation unit 901, a region-specific parameter setting unit 902, and a region-specific image processing execution unit 903 are provided. The surveillance camera 601 has a region-specific estimate value calculation unit 901, a camera setting acquisition unit 708, a threshold calculation unit 709, a region-specific parameter setting unit 902, a region-specific image processing execution unit 903, an encoder unit 712, and a network I/F 713.

Figure 10 is a flowchart showing a processing method of the surveillance camera 601 according to the second embodiment. First, in step S1001, the area-specific estimated value calculation unit 901 estimates the degree of change in aging for each area of the photoelectric conversion element 100 having the APD 201, and calculates an estimated value of the degree of change for each area of the photoelectric conversion element 100. For example, the DCR of the photoelectric conversion element 100 changes as the usage time increases, and the count value output changes. The estimated value calculation unit 707 calculates an estimated value of the degree of change for each area based on the number of avalanche emissions since the start of operation, the exposure conditions, the temperature, or the luminance information of the subject. The unit of the area may be a pixel unit, or a divided area unit prepared to obtain evaluation values such as AF (Auto Focus) and AE (Auto Exposure).

Next, in step S1002, the camera setting acquisition unit 708 acquires camera setting information from the imaging system control unit 705 and the control unit 706. The camera setting information is, for example, on/off information for a dark area correction function executed by the region-specific image processing execution unit 903. The surveillance camera 601 has a function for improving the visibility of dark subjects by raising the gamma of low-luminance areas. The dark area correction by the region-specific image processing execution unit 903 improves the visibility of subjects in dark areas, but there is a commonly known trade-off that makes noise and defective pixel noise more noticeable. In this embodiment, a method for calculating a change threshold for determining an estimated change degree value for each region will be described, taking into account the above-mentioned trade-off.

In steps S1003 to S1005, the threshold calculation unit 709 calculates a change threshold for determining the degree of change for each region based on the camera setting information described above. First, in step S1003, the threshold calculation unit 709 determines whether the dark compensation function is on or off based on the on/off information of the dark compensation function in the camera setting information. If the dark compensation function is on, processing proceeds to step S1004, and if the dark compensation function is off, processing proceeds to step S1005.

In step S1004, the threshold calculation unit 709 sets a small fifth value as the change threshold. Then, the process proceeds to step S1006.

In step S1005, the threshold calculation unit 709 sets a sixth value greater than the fifth value as the change threshold. Then, the process proceeds to step S1006.

As described above, in steps S1003 to S1005, when the dark compensation function is on, the change threshold is set small to make it easier to correct noise and defective pixel noise that are easily made apparent by dark compensation. On the other hand, when the dark compensation function is off, the change threshold is set large to reduce defective pixel noise and noise more than necessary, so as not to lose edge information of the subject.

Next, in step S1006, the region-specific parameter setting unit 902 determines whether the change degree estimate for each region calculated by the region-specific estimate calculation unit 901 is equal to or greater than the change threshold set by the threshold calculation unit 709. For regions where the change degree estimate is equal to or greater than the change threshold, processing proceeds to step S1007. For regions where the change degree estimate is not equal to or greater than the change threshold, processing proceeds to step S1008.

In step S1007, the region-specific parameter setting unit 902 sets image processing parameters for the corresponding region to strongly reduce noise and defective pixel noise. Then, the process proceeds to step S1009.

In step S1008, the area-specific parameter setting unit 902 sets image processing parameters for the corresponding area to leave noise and defective pixel noise as they are. Then, the process proceeds to step S1009.

Finally, in step S1009, the region-specific image processing execution unit 903 executes image processing for each region on the count value output by the photoelectric conversion element 100 based on the image processing parameters for each region set by the parameter setting unit 710.

As described above, the surveillance camera 601 calculates the change threshold taking into account the effects of noise and defective pixel noise emphasized by the image processing of the regional image processing execution unit 903, and sets optimal image processing parameters for each region according to the estimated degree of change and the change threshold for each region. Therefore, the surveillance camera 601 is able to optimally set the image processing parameters of the regional image processing execution unit 903 for each region in response to aging (increase in DCR noise and defective pixel noise) that occurs in the photoelectric conversion element 100 having the APD 201.

In this embodiment, a configuration has been described in which setting information for dark area correction processing is used as the camera setting information acquired by the camera setting acquisition unit 708. However, the camera setting information is not limited to this. For example, the camera setting information may be setting information for codec, gradation correction, noise reduction processing, defective pixel correction, demosaicing processing, sharpness processing, or saturation enhancement processing. In addition, the image processing performed by the image processing execution unit 711 is gradation correction, noise reduction processing, defective pixel correction, demosaicing processing, sharpness processing, saturation enhancement processing, dark area correction, etc.

In the above-described embodiment, a so-called lens-integrated imaging device in which the optical system 701 is integrally formed with the surveillance camera 601 has been described as an example of an imaging device, but the present invention is not limited to this. For example, a so-called lens-interchangeable imaging device in which the surveillance camera 601 and a lens unit equipped with the optical system 701 are separately provided may also be used.

In addition, in the above-described embodiment, the surveillance camera 601 has been described as an example of an imaging device, but the present invention is not limited to this. For example, the present invention may be configured to employ imaging devices other than the surveillance camera 601, such as portable devices such as digital cameras, digital video cameras, and smartphones, or wearable devices.

In addition, in the above-described embodiment, the client device 603, which is an information processing device, is, for example, an electronic device such as a personal computer (PC), but is not limited to this. For example, the client device 603 may be configured to employ other electronic devices, such as a smartphone or a tablet terminal.

Other Embodiments
The present disclosure can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

The above-mentioned embodiments are merely illustrative examples of how the present disclosure may be implemented, and the technical scope of the present disclosure should not be interpreted in a limiting manner. In other words, the present disclosure may be implemented in various forms without departing from its technical concept or main features.

The disclosure of this embodiment includes the following configuration, method, and program.
(Configuration 1)
an imaging section including an avalanche photodiode;
an estimate calculation unit that calculates an estimate of a degree of change of the imaging unit;
an image processing unit that performs image processing on an output signal of the imaging unit based on the change degree estimated value and an image processing parameter based on a first threshold value;
(Configuration 2)
The imaging unit includes:
The avalanche photodiode;
2. The imaging device according to claim 1, further comprising a counter for counting a signal from the avalanche photodiode.
(Configuration 3)
the imaging unit further includes a shaping unit that shapes a waveform of a signal from the avalanche photodiode;
3. The imaging device according to claim 2, wherein the counter counts the signal shaped by the shaping unit.
(Configuration 4)
The imaging device described in any one of configurations 1 to 3, characterized in that the estimated value calculation unit calculates an estimated value of the degree of change of the imaging unit based on the number of avalanche flashes since the start of operation, exposure conditions, temperature, or brightness information of the subject.
(Configuration 5)
The imaging device according to any one of configurations 1 to 4, wherein the estimated value calculation unit calculates a cumulative value of luminance based on an output signal of the imaging unit, and calculates an estimated value of a degree of change of the imaging unit based on the cumulative value of luminance.
(Configuration 6)
6. The imaging device according to any one of configurations 1 to 5, further comprising a threshold setting section that sets the first threshold on the basis of setting information of the imaging device.
(Configuration 7)
The imaging device according to any one of configurations 1 to 6, characterized in that the setting information of the imaging device is setting information of a codec, a gradation correction, a noise reduction process, a defective pixel correction, a demosaicing process, a sharpness process, a saturation enhancement process, or a dark area correction.
(Configuration 8)
the setting information of the imaging device includes a bit rate of an encoder;
The imaging device described in configuration 6, characterized in that the threshold setting unit sets a first value as the first threshold when the bit rate is equal to or greater than a second threshold, and sets a second value larger than the first value as the first threshold when the bit rate is not equal to or greater than the second threshold.
(Configuration 9)
the setting information of the imaging device includes a Qp value of an encoder;
The imaging device described in configuration 6, characterized in that when the Qp value is equal to or less than a third threshold, the threshold setting unit sets a third value as the first threshold, and when the Qp value is not equal to or less than the third threshold, sets a fourth value greater than the third value as the first threshold.
(Configuration 10)
The setting information of the imaging device includes an encoding method, a bit rate, and a Qp value of an encoder;
The threshold setting unit is
if the encoding method is VBR encoding and the bit rate is equal to or greater than a second threshold, set the first threshold to a first value;
if the encoding method is VBR encoding and the bit rate is not equal to or greater than a second threshold, set the first threshold to a second value greater than the first value;
if the encoding method is CBR encoding and the Qp value is equal to or smaller than a third threshold, set the first threshold to a third value;
The imaging device according to configuration 6, characterized in that, when the encoding method is CBR encoding and the Qp value is not equal to or less than a third threshold, a fourth value greater than the third value is set as the first threshold.
(Configuration 11)
The setting information of the imaging device is on/off information of a dark compensation function,
The imaging device according to configuration 6, wherein the threshold setting unit sets a fifth value as the first threshold when the dark compensation function is on, and sets a sixth value larger than the fifth value as the first threshold when the dark compensation function is off.
(Configuration 12)
a parameter setting unit that sets a first image processing parameter when the change degree estimation value is equal to or greater than the first threshold, and sets a second image processing parameter when the change degree estimation value is not equal to or greater than the first threshold,
The imaging device according to any one of configurations 1 to 11, characterized in that the image processing unit performs image processing based on a first image processing parameter or a second image processing parameter set by the parameter setting unit.
(Configuration 13)
13. The imaging apparatus according to configuration 12, wherein the first image processing parameter is a parameter for performing processing for reducing noise or defective pixel noise more strongly than the second image processing parameter.
(Configuration 14)
The imaging device according to any one of configurations 1 to 13, characterized in that the image processing performed by the image processing unit is gradation correction, noise reduction processing, defective pixel correction, demosaicing processing, sharpness processing, saturation enhancement processing, or dark area correction.
(Configuration 15)
the estimate calculation unit calculates a change degree estimate value for each region of the imaging unit;
The imaging device described in any one of configurations 1 to 14, characterized in that the image processing unit performs image processing for each region on the output signal of the imaging unit based on an estimated value of the degree of change for each region and an image processing parameter for each region that is based on a first threshold value.
(Method 1)
an estimated value calculation step of calculating an estimated value of a degree of change of an imaging unit including an avalanche photodiode;
a step of performing image processing on an output signal of the imaging unit based on the estimated value of degree of change and an image processing parameter based on a first threshold value, the image processing step comprising:
(Program 1)
an estimated value calculation step of calculating an estimated value of a degree of change of an imaging unit including an avalanche photodiode;
and an image processing step of performing image processing on an output signal of the imaging unit based on the change degree estimated value and an image processing parameter based on a first threshold value.

100 Photoelectric conversion element, 601 Surveillance camera, 701 Optical system, 702 CPU, 703 ROM, 704 RAM, 705 Imaging system control unit, 706 Control unit, 707 Estimated value calculation unit, 708 Camera setting acquisition unit, 709 Threshold calculation unit, 710 Parameter setting unit, 711 Image processing execution unit, 712 Encoder unit, 713 Network I/F

Claims (17)

an imaging section including an avalanche photodiode;
an estimate calculation unit that calculates an estimate indicating a degree of deterioration of the imaging unit based on at least one of information regarding the number of avalanche emissions in the avalanche photodiode, an exposure condition of the imaging unit, and a temperature of the imaging unit;
an image processing unit that performs image processing on an output signal of the imaging unit based on the estimated value indicating a degree of deterioration and an image processing parameter based on a first threshold value; The imaging unit includes:
The avalanche photodiode;
2. The image pickup apparatus according to claim 1, further comprising a counter for counting the signal from the avalanche photodiode. the imaging unit further includes a shaping unit that shapes a waveform of a signal from the avalanche photodiode;
3. The imaging device according to claim 2, wherein the counter counts the signal shaped by the shaping unit. The imaging device according to claim 1 , wherein the estimated value calculation unit calculates an estimated value indicating a degree of deterioration of the imaging unit based on the number of avalanche flashes since the start of operation, exposure conditions, temperature, or subject brightness information. The imaging device according to claim 1, characterized in that the estimate value calculation unit calculates a cumulative value of luminance of multiple images obtained based on the output signal of the imaging unit, and calculates an estimate value indicating a degree of deterioration of the imaging unit based on the cumulative value of luminance. The imaging device according to claim 1, further comprising a threshold setting unit that sets the first threshold based on setting information of the imaging device. The imaging device according to claim 6 , wherein the setting information of the imaging device is codec setting information. the setting information of the imaging device includes a bit rate of an encoder;
7. The imaging device according to claim 6, wherein the threshold setting unit sets a first value as the first threshold when the bit rate is equal to or greater than a second threshold, and sets a second value greater than the first value as the first threshold when the bit rate is not equal to or greater than the second threshold. the setting information of the imaging device includes a Qp value of an encoder;
7. The imaging device according to claim 6, wherein the threshold setting unit sets a third value as the first threshold when the Qp value is equal to or less than a third threshold, and sets a fourth value greater than the third value as the first threshold when the Qp value is not equal to or less than the third threshold. The setting information of the imaging device includes an encoding method, a bit rate, and a Qp value of an encoder;
The threshold setting unit is
if the encoding method is VBR encoding and the bit rate is equal to or greater than a second threshold, set the first threshold to a first value;
if the encoding method is VBR encoding and the bit rate is not equal to or greater than a second threshold, set the first threshold to a second value greater than the first value;
if the encoding method is CBR encoding and the Qp value is equal to or smaller than a third threshold, set the first threshold to a third value;
7. The imaging device according to claim 6, wherein when the encoding method is CBR encoding and the Qp value is not equal to or less than a third threshold, a fourth value greater than the third value is set as the first threshold. The setting information of the imaging device is on/off information of a dark compensation function,
7. The imaging device according to claim 6, wherein the threshold setting unit sets a fifth value as the first threshold when the dark compensation function is on, and sets a sixth value larger than the fifth value as the first threshold when the dark compensation function is off. a parameter setting unit that sets a first image processing parameter when the estimated value indicating the degree of deterioration is equal to or greater than the first threshold, and sets a second image processing parameter when the estimated value indicating the degree of deterioration is not equal to or greater than the first threshold,
2. The imaging apparatus according to claim 1, wherein the image processing unit performs image processing based on a first image processing parameter or a second image processing parameter set by the parameter setting unit. The imaging device according to claim 12, characterized in that the first image processing parameter is a parameter for performing processing to reduce noise or defective pixel noise more strongly than the second image processing parameter. 2. The imaging apparatus according to claim 1, wherein the image processing performed by the image processing unit is noise reduction processing or defective pixel correction processing . the estimated value calculation unit calculates an estimated value indicating a degree of deterioration for each region of the imaging unit;
The imaging device according to claim 1, characterized in that the image processing unit performs image processing for each region on the output signal of the imaging unit based on an estimated value indicating the degree of deterioration for each region and an image processing parameter for each region that is based on a first threshold value. an estimate calculation step of calculating an estimate indicating a degree of deterioration of an imaging section including an avalanche photodiode based on at least one of avalanche light emission counts in the avalanche photodiode, exposure conditions of the imaging section, and temperature;
A processing method for an imaging device, comprising: an image processing step of performing image processing on an output signal of the imaging unit based on an image processing parameter based on the estimated value indicating the degree of deterioration and a first threshold value. an estimate calculation step of calculating an estimate indicating a degree of deterioration of an imaging section including an avalanche photodiode based on at least one of avalanche light emission counts in the avalanche photodiode, exposure conditions of the imaging section, and temperature;
and an image processing step of performing image processing on an output signal of the imaging unit based on the estimated value indicating the degree of deterioration and an image processing parameter based on a first threshold value.

Images