JP7779020B2 - Image processing device and image sensor - Google Patents

Description

The present invention relates to an image processing device and an imaging element.

Patent Document 1 describes a tone mapping technique for reproducing real-world scenes into high dynamic range images.
[Prior art documents]
[Patent Documents]
[Non-patent Document 1] Erik Reinhard, et al. , "Photographic Tone Reproduction for Digital Images", SIGGRAPH '02: Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 2002, pp. 267-276, https://doi. org/10.1145/566570.566575

A first aspect of the present invention provides an image processing device. The image processing device includes a signal processing unit that generates an integrated conversion function that combines multiple conversion functions, including a first conversion function that converts the input luminance of a pixel signal to an output luminance, and a second conversion function that converts the input luminance to an output luminance different from the output luminance, and converts the luminance of the pixel signal using the integrated conversion function.

A second aspect of the present invention provides an image processing device. The image processing device includes a signal processing unit that converts the luminance of a pixel signal using a conversion function that converts the input luminance of the pixel signal to an output luminance, and the conversion function has three or more inflection points.

A third aspect of the present invention provides an imaging element. The imaging element includes a plurality of pixels that output pixel signals in response to incident light, and a signal processing unit that performs the necessary arithmetic processing for image processing on the pixel signals. The signal processing unit generates an integrated conversion function that combines a plurality of conversion functions, including a first conversion function that converts input luminance to output luminance and a second conversion function that converts input luminance to an output luminance different from the output luminance, and converts the luminance of the pixel signals using the integrated conversion function.

Note that the above summary of the invention does not list all of the necessary features of the present invention. Subcombinations of these features may also constitute inventions.

FIG. 1 is a cross-sectional view of an image sensor 100 according to the present embodiment. 1 is a diagram illustrating the pixel array and blocks 131 of the imaging chip 113. FIG. FIG. 10 is a circuit diagram corresponding to a block 131 of the imaging chip 113. FIG. 2 is a block diagram showing the functional configuration of the image sensor 100. FIG. 2 is a block diagram showing the functional configuration of the image sensor 100. FIG. 1 is a block diagram showing the configuration of an imaging device 500 according to the present embodiment. FIG. 10 is a diagram illustrating an example of a conversion function. FIG. 10 is a flowchart illustrating an example of luminance conversion. 10A and 10B are diagrams illustrating an example of a step of determining a reference luminance and a band from a luminance histogram. FIG. 10 is a diagram showing an example of a captured image in which luminance has been converted using a conversion function according to the prior art. FIG. 10 is a diagram showing an example of a captured image in which luminance has been converted using a conversion function according to the prior art. FIG. 10 is a diagram showing an example of a captured image in which luminance has been converted using an integrated conversion function according to the present embodiment. FIG. 10 is a diagram showing an example of an image in which luminance has been converted in consideration of feature points. FIG. 10 is a diagram illustrating an example of a saliency map. 10A and 10B are diagrams illustrating another example of the step of determining the reference luminance and the band from the luminance histogram.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

Figure 1 is a cross-sectional view of an image sensor 100 according to this embodiment. The image sensor 100 comprises an image sensor chip 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The image sensor chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to each other by bumps 109, which are electrically conductive connecting parts made of Cu or the like.

As shown in the figure, incident light is mainly incident in the positive Z-axis direction indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which incident light is incident is referred to as the back surface. As shown by the coordinate axes, the left direction on the paper, perpendicular to the Z-axis, is the positive X-axis direction, and the front direction on the paper, perpendicular to the Z-axis and X-axis, is the positive Y-axis direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood, based on the coordinate axes in Figure 1.

An example of the imaging chip 113 is a back-illuminated MOS image sensor. The PD layer 106 is arranged on the back side of the wiring layer 108. The PD layer 106 has multiple PDs (photodiodes) 104 arranged two-dimensionally and transistors 105 provided corresponding to the PDs 104.

A color filter 102 is provided on the incident light side of the PD layer 106 via a passivation film 103. There are multiple types of color filters 102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A combination of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

A microlens 101 is provided on the incident light side of the color filter 102, corresponding to each pixel. The microlens 101 focuses the incident light toward the corresponding PD 104.

The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered and may include passive and active elements.

A number of bumps 109, which serve as connection sections, are arranged on the surface of the wiring layer 108. These bumps 109 are aligned with and joined to a number of bumps 109 provided on the opposing surface of the signal processing chip 111, thereby electrically connecting the imaging chip 113 and the signal processing chip 111.

Similarly, a plurality of bumps 109, which serve as connecting portions, are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned and joined to each other, thereby electrically connecting the signal processing chip 111 and the memory chip 112.

The imaging element 100 is formed by bonding the imaging chip 113, signal processing chip 111, and memory chip 112 together in a wafer state before they are separated into chips, and then dicing the bonded wafer.

When bonding wafers together, plasma is applied to the wafer surfaces using an activation device to activate the bonding surfaces of the wafers. The wafers with activated surfaces are then bonded together through hydrogen bonds, van der Waals bonds, and covalent bonds that form when they come into contact, forming a laminated substrate. If the two wafers are hydrogen bonded through their contact with each other, after forming the laminated substrate, the laminated substrate is placed in a heating device such as an annealing furnace and heated to create covalent bonds between the wafers.

Note that activation includes treating the bonding surface of at least one of the substrates so that when the bonding surface of one wafer comes into contact with the bonding surface of another wafer, hydrogen bonds, van der Waals bonds, covalent bonds, etc. are formed, resulting in solid-state bonding without melting. In other words, activation includes creating dangling bonds on the wafer surface, making it easier to form bonds.

More specifically, in an activation device, oxygen gas, which is the process gas, is excited to form plasma in a reduced-pressure atmosphere, and oxygen ions are irradiated onto the surfaces that will become the bonding surfaces of the two substrates. For example, if the wafer is a substrate with an SiO film formed on Si, the irradiation of these oxygen ions breaks the SiO bonds on the wafer surface that will become the bonding surface during stacking, forming dangling bonds of Si and O. The formation of such dangling bonds on the wafer surface is sometimes referred to as activation.

If a substrate with dangling bonds is exposed to the atmosphere, for example, moisture in the air will bind to the dangling bonds, and the substrate surface will become covered with hydroxyl groups (OH groups). The substrate surface will then become hydrophilic, making it more susceptible to bonding with water molecules. In other words, activation results in the substrate surface becoming hydrophilic. Furthermore, in solid-state bonding, the presence of impurities such as oxides at the bonding interface, as well as defects at the bonding interface, affect the bonding strength. Therefore, cleaning the bonding surface can be considered part of the activation process.

Furthermore, the wafer can be activated by applying pure water or the like to the hydrophilized surface of the wafer that will become the bonding surface using an apparatus not shown. This causes the wafer surface to have OH groups attached, i.e., be terminated with OH groups.

By heating the laminated substrate, the bumps 109 on each of the bonding surfaces of the two wafers are integrated with each other, forming an electrical connection between the wafers. By forming the bumps 109 from a material that melts at a low temperature, such as indium or a tin-silver alloy, the laminated substrate can be reflow processed at a low temperature of 200°C or less. Alternatively, if the bumps 109 are made from a conductive metal such as copper, they expand during the heat treatment, pressuring the bumps 109 between the wafers together and bonding them through solid-state diffusion.

The bonding between the bumps 109 is not limited to Cu bump bonding using solid-phase diffusion, but micro-bump bonding using solder melting may also be used. Furthermore, it is sufficient to provide approximately one bump 109 for each pixel block, as described below. Therefore, the size of the bumps 109 may be larger than the pitch of the PDs 104. Furthermore, in peripheral regions other than the pixel region where the pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel region may also be provided.

The signal processing chip 111 has TSVs (through silicon vias) 110 that connect the circuits provided on the front and back surfaces. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and in the memory chip 112. The TSVs 110 may also be used to electrically connect the circuits provided on the memory chip 112 and the circuits provided on the imaging chip 113.

In this way, the imaging chip 113 and the signal processing chip 111 are bonded to each other via their opposing surfaces and the bumps 109. The signal processing chip 111 and the memory chip 112 are also bonded to each other via their opposing surfaces and connected to each other via the bumps 109 and the TSVs 110 provided in the signal processing chip 111. Note that the signal processing chip 111 and the memory chip 112 may also be bonded to each other via their opposing surfaces and connected to each other via at least one of the bumps 109 and the TSVs 110.

Figure 2 is a diagram illustrating the pixel array and blocks 131 of the imaging chip 113. In particular, it shows the imaging chip 113 as viewed from the back side. More than 20 million pixels are arranged in a matrix in the pixel area. These pixels are divided into blocks containing at least two pixels. In this embodiment, 16 pixels (4 x 4 adjacent pixels) form one block. The grid lines in the diagram illustrate the concept of adjacent pixels collectively forming block 131.

As shown in the enlarged partial view of the pixel area, block 131 contains four so-called Bayer arrays, arranged vertically and horizontally, each consisting of four pixels: green pixels Gb and Gr, blue pixel B, and red pixel R. The green pixels are pixels that have a green filter as their color filter 102 and receive light in the green wavelength band of incident light. Similarly, the blue pixels are pixels that have a blue filter as their color filter 102 and receive light in the blue wavelength band, and the red pixels are pixels that have a red filter as their color filter 102 and receive light in the red wavelength band.

Figure 3 is a circuit diagram corresponding to block 131 of the imaging chip 113. In the figure, a rectangle surrounded by a dotted line typically represents a circuit corresponding to one pixel. Note that at least some of the transistors described below correspond to transistor 105 in Figure 1.

As described above, block 131 is formed from 16 pixels. The 16 PDs 104 corresponding to each pixel are connected to transfer transistors 302, and the gates of each transfer transistor 302 are connected to TX wiring 307, which supplies transfer pulses. In this embodiment, the TX wiring 307 is commonly connected to all 16 transfer transistors 302.

The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and the so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the amplification transistor 304. The drain of the reset transistor 303 is connected to the Vdd wiring 310 to which the power supply voltage is supplied, and its gate is connected to the reset wiring 306 to which a reset pulse is supplied. In this embodiment, the reset wiring 306 is commonly connected to all 16 reset transistors 303.

The drain of each amplification transistor 304 is connected to a Vdd wiring 310 to which a power supply voltage is supplied. The source of each amplification transistor 304 is connected to the drain of the corresponding selection transistor 305. The gate of each selection transistor is connected to a decoder wiring 308 to which a selection pulse is supplied. In this embodiment, the decoder wiring 308 is provided independently for each of the 16 selection transistors 305. The sources of each selection transistor 305 are connected to a common output wiring 309. A load current source 311 supplies current to the output wiring 309. In other words, the output wiring 309 for the selection transistor 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or the signal processing chip 111 side.

Here, we will explain the process from the start of pixel exposure to the output of a pixel signal after exposure is complete. When a reset pulse is applied to the reset transistor 303 via the reset wiring 306 and, at the same time, a transfer pulse is applied to the transfer transistor 302 via the TX wiring 307, the potentials of the PD 104 and floating diffusion FD are reset, and exposure begins.

When the transfer pulse application is stopped, PD 104 converts the incident light it receives into electric charge and accumulates it. Then, when the transfer pulse is applied again without the reset pulse being applied, exposure ends. The electric charge accumulated up until the end of exposure is transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after exposure ends. Then, when a selection pulse is applied to the selection transistor 305 via the decoder wiring 308, the fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, a pixel signal corresponding to the reset potential and signal potential is output from the unit pixel to the output wiring 309.

As shown in the figure, in this embodiment, the reset wiring 306 and TX wiring 307 are common to the 16 pixels that form block 131. In other words, the reset pulse and transfer pulse are each applied simultaneously to all 16 pixels. Therefore, all pixels that form block 131 start and end exposure at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output to the output wiring 309 by sequentially applying selection pulses to each selection transistor 305.

By configuring the circuit in this way based on block 131, it is possible to control the exposure time for each block 131. Because the exposure time can be controlled for each block, adjacent blocks 131 can output pixel signals with different exposure times. Furthermore, it is possible to set a common exposure time for all blocks 131, and perform one exposure and pixel signal output for a certain block 131, while repeating two exposures and pixel signal outputs for an adjacent block 131. This latter type of repeated control of exposure and pixel signal output using a common unit time is called unit time control. Note that when performing unit time control, if the start and end points of exposure are synchronized for all blocks 131, the reset wiring 306 may be commonly connected to all reset transistors 303 on the imaging chip 113.

Figure 4 is a block diagram showing the functional configuration of the image sensor 100. Here, we will particularly explain the flow of pixel signals.

The analog multiplexer 411 sequentially selects the 16 PDs 104 that make up the block 131 and outputs each pixel signal to the output wiring 309. The multiplexer 411, together with the PDs 104, is formed on the imaging chip 113.

The pixel signals output via the multiplexer 411 undergo correlated double sampling (CDS) and analog-to-digital (A/D) conversion by a signal processing circuit 412 formed in the signal processing chip 111, which performs CDS and analog-to-digital (A/D) conversion. The A/D conversion converts the input analog pixel signals into 12-bit digital pixel signals. The A/D converted pixel signals are passed to an arithmetic circuit 415, also formed in the signal processing chip 111. The arithmetic circuit 415 performs the necessary arithmetic processing on the received pixel signals for subsequent image processing, and passes them to the demultiplexer 413.

The demultiplexer 413 stores the received pixel signals in the pixel memories 414 corresponding to each pixel. Each pixel memory 414 has the capacity to store the pixel signals after arithmetic processing has been performed. The demultiplexer 413 and pixel memories 414 are formed on the memory chip 112.

The arithmetic circuit 415 reads pixel signals to be used for arithmetic processing from the pixel memory 414 via the demultiplexer 413. Alternatively, in response to an external transfer request, the arithmetic circuit 415 transfers the pixel signals read from the pixel memory 414 via the demultiplexer 413 to a downstream image processing unit. The arithmetic circuit 415 may also be provided in the memory chip 112.

Also, while the diagram shows the flow of pixel signals for one block, in reality these exist for each block and operate in parallel. However, an arithmetic circuit 415 does not have to exist for each block; for example, one arithmetic circuit 415 may process sequentially while referencing the values in the pixel memory 414 corresponding to each block in order.

Figure 5 is a block diagram showing the functional configuration of the image sensor 100. Here, we will mainly explain the specific configuration of the signal processing chip 111 and the setting unit 460 provided in the signal processing chip 111.

The signal processing chip 111 includes a sensor control unit 441, a synchronization control unit 443, and a signal control unit 444 as distributed control functions, as well as a drive control unit 420 that controls these control units. The drive control unit 420 is a control circuit that converts instructions from the system control unit 501, which is responsible for the overall control of the imaging device, into control signals that can be executed by each control unit and passes them on to each unit.

The sensor control unit 441 is responsible for controlling the transmission of control pulses related to the charge accumulation and charge readout of each pixel, which are sent to the imaging chip 113. Specifically, the sensor control unit 441 controls the start and end of exposure by sending reset pulses and transfer pulses to the target pixels, and outputs pixel signals to the output wiring 309 by sending selection pulses to the readout pixels.

The synchronization control unit 443 sends a synchronization signal to the imaging chip 113. Each pulse becomes active in the imaging chip 113 in synchronization with the synchronization signal. For example, by adjusting the synchronization signal, random control, thinning control, etc., can be realized, which controls only specific pixels belonging to the same block 131.

The signal control unit 444 is mainly responsible for timing control of the A/D converter 412b. The pixel signals output via the output wiring 309 are input to the CDS circuit 412a and A/D converter 412b via the multiplexer 411. The A/D converter 412b is controlled by the signal control unit 444 and converts the input pixel signals into digital signals. The converted digital pixel signals are passed to the arithmetic circuit 415 for arithmetic processing. The processed pixel signals are passed to the demultiplexer 413 of the memory chip 112 and stored as digital pixel values in the pixel memories 414 corresponding to each pixel.

The drive control unit 420 sends a control signal to the sensor control unit 441 according to the exposure conditions for each block 131. The exposure conditions are conditions that change the brightness of the image to be captured, such as exposure time, aperture value, and ISO sensitivity. The drive control unit 420 may set the exposure conditions for each block 131 according to the luminance distribution of the scene acquired from the system control unit 501, or may acquire the exposure conditions for each block 131 from the system control unit 501.

In response to a transfer request from the system control unit 501, the drive control unit 420 reads the target pixel signal from the pixel memory 414 via the arithmetic circuit 415 and demultiplexer 413 and transfers it to the image processing unit 511 of the system control unit 501. The pixel memory 414 is provided with a data transfer interface that transmits the pixel signal in response to the transfer request. The data transfer interface is connected to a data transfer line that connects to the image processing unit 511. The data transfer line is formed, for example, by a data bus among the bus lines. In this case, the transfer request from the system control unit 501 to the drive control unit 420 is executed by address specification using the address bus.

Transmission of pixel signals via the data transfer interface is not limited to the addressing method, and various other methods can be used. For example, a double data rate method can be used, which uses both the rising and falling edges of the clock signal used to synchronize each circuit when transferring data. A burst transfer method can also be used, which transfers data all at once by omitting some steps such as addressing, thereby increasing speed. It is also possible to use a combination of a bus method using lines connecting the control unit, memory unit, and input/output unit in parallel, and a serial method which transfers data serially one bit at a time.

By configuring it in this way, the image processing unit 511 can receive only the necessary pixel signals, allowing image processing to be completed quickly, especially when forming low-resolution images.

The signal processing chip 111 further includes a setting unit 460 that sets exposure conditions for each block 131. The setting unit 460 acquires the luminance distribution of the scene from the system control unit 501 and sets exposure conditions for each block 131 according to the luminance distribution. Exposure conditions are conditions that change the brightness of the acquired image, such as exposure time, aperture value, ISO sensitivity, etc. The setting unit 460 also determines the timing of opening and closing the shutter according to the exposure time of each block 131.

By providing the setting unit 460 on the same signal processing chip 111 as the drive control unit 420, which controls each control unit, high-speed processing can be achieved. Furthermore, rather than providing the setting unit 460 separately from the drive control unit 420, the drive control unit 420 may take on the functions of the setting unit 460.

Furthermore, the setting unit 460 may be provided on the memory chip 112 instead of the signal processing chip 111. If the setting unit 460 is provided on the memory chip 112, transmission speeds can be increased via the bumps 109 and TSVs 110, while more space can be secured on the signal processing chip 111.

Figure 6 is a block diagram showing the configuration of an image capture device 500 according to this embodiment. The image capture device 500 mainly includes an image sensor 100, a photographing lens 520, a system control unit 501, a photometry unit 503, a work memory 504, a recording unit 505, and a display unit 506.

The photometry unit 503 detects the luminance distribution of the scene prior to a series of shooting sequences that generate image data. The photometry unit 503 includes, for example, an AE sensor with approximately 1 million pixels.

The photographing lens 520 guides the subject light beam incident along the optical axis O to the image sensor 100. The photographing lens 520 is an example of an optical system. The photographing lens 520 is composed of a group of multiple optical lenses, and focuses the subject light beam from the scene near its focal plane. Note that in Figure 6, the photographing lens 520 is represented by a single virtual lens placed near the pupil. The photographing lens 520 may also be an interchangeable lens that can be attached to and detached from the image capture device 500.

The image processing unit 511 receives pixel signals from the drive control unit 420 of the image sensor 100. The image processing unit 511 performs various image processing using the work memory 504 as a workspace to generate image data. For example, the image processing unit 511 detects feature points from multiple captured images and synthesizes the multiple images based on the feature points to generate final image data. When generating image data in JPEG file format, the image processing unit 511 performs white balance processing, gamma processing, etc., before performing compression processing. The generated image data is recorded in the recording unit 505 and converted into a display signal that is displayed on the display unit 506 for a predetermined time. The image processing unit 511 is an example of a signal processing unit that generates a luminance conversion function, and may be configured as an ASIC independent of the system control unit 501, or may be provided on the memory chip 112.

The system control unit 501 is responsible for the integrated control of the entire imaging device. Upon receiving instructions from the user, the system control unit 501 generates imaging instructions to be sent to the image sensor 100. Furthermore, the calculation unit 512 of the system control unit 501 receives the output of the photometry unit 503 and calculates the luminance for each region of the scene. The calculation unit 512 outputs the luminance distribution of the scene to the setting unit 460 described above. The calculation unit 512 also performs various calculations to operate the imaging device 500.

The calculation unit 512 may perform the functions of the setting unit 460 described above. In this case, the calculation unit 512 sets exposure conditions for each block 131 according to the luminance distribution of the scene. The calculation unit 512 also determines the timing of opening and closing the shutter according to the exposure conditions set for each block 131.

Figure 7 shows an example of a conversion function. Before explaining Figure 7, we will explain the conversion function used in the tone mapping process, which converts the luminance of a real-world scene into the luminance for an image.

The conversion function is an increasing function that converts input luminance into a smaller output luminance based on a reference luminance. The conversion function G(x) is a sigmoid function. The conversion function converts input luminance into output luminance based on the reference luminance, the average luminance of pixel signals in the region where the pixel is located, and a predetermined brightness adjustment parameter. For example, the Reinhard conversion function is known as such a conversion function.

The reference luminance Lm(x) of a pixel at coordinate x is expressed by the above formula. The reference luminance is the luminance obtained by scaling the input luminance Lw(x). Here, α is a brightness adjustment parameter, and _Lw,Global is the average luminance of the entire image (global average luminance). The brightness adjustment parameter α is a value in the range of 0<α<1, and different values can be set depending on the brightness of the entire image. For example, to maintain the image at a bright gradation, the brightness adjustment parameter α is changed from 0.18 to 0.36 and 0.72, and to maintain the image at a dark gradation, the brightness adjustment parameter α is changed from 0.18 to 0.09 and 0.045.

The output luminance Ld(x) is expressed by the following formula:

Here, L _m,Local (x) is the average luminance (local average luminance) in the region where the pixel is located. Putting these equations together, the conversion function that converts input luminance to output luminance is as follows:

Here, G(x) is the conversion function, X(x) is the input luminance of each RGB value, and X'(x) is the output luminance of each RGB value. These formulas for the conversion function G(x) can be summarized as follows:

As can be seen from the above formula, the conversion function G(x) is determined by the global average luminance L _w,Global , the local average luminance, and the brightness adjustment parameter α. Here, since the global average luminance L _w,Global and the local average luminance are preset values, it can be seen that the brightness adjustment parameter α is the only parameter for adjusting the luminance band of the conversion result.

Next, referring to Figure 7, the conversion functions s1, s2, s3, and s123 that convert input luminance Lw to output luminance Ld will be explained. The horizontal axis represents the luminance of input luminance Lw, and the vertical axis represents the luminance of output luminance Ld. Each of the conversion functions s1, s2, and s3 is a function that uses one reference luminance as a base, and draws an S-shaped curve with two inflection points before and after the rising edge. Conversion function s2 uses the overall average luminance as the reference luminance. Conversion functions s1 and s3 use luminances lower and higher than the overall average luminance as their reference luminances, respectively, and place emphasis on luminance bands darker and brighter than the overall average luminance.

In both conversion functions, when the input luminance Lw is within a certain range centered on the reference luminance, the output luminance Ld functions as an increasing function that varies between 0 and 1. Outside this range, however, the output luminance Ld is compressed to a value of either 0 or 1, and the input luminance Lw is essentially ignored. Figure 7 shows the range in which the input luminance Lw is ignored in conversion function s2.

On the other hand, the conversion function s123 is an integrated conversion function that combines the conversion functions s1, s2, and s3 according to the following formula.

Ld(x) is the output luminance obtained by an integrated conversion function that combines N conversion functions. _Lm,n (x) is the reference luminance of each conversion function, where n = 1, ..., N (here, N = 3 because three conversion functions s1, s2, and s3 are combined). _Lm,Local,n (x) is the local average luminance of each conversion function.
k _n is the combination ratio of the conversion function,
is.

As shown in Figure 7, the conversion function s123 combines multiple conversion functions with different reference luminances, and therefore has three or more inflection points. Using this conversion function s123, a wider range of input luminance Lw can be converted without compression, allowing the luminance distribution of real-world scenes to be reproduced more accurately.

Figure 8 is a flow diagram showing an example of brightness conversion. Note that in the explanation of Figure 8, the image processing unit 511 is described as the main operator of each step, but this is not limited to this. In other examples, the setting unit 460 and arithmetic circuit 415 of the image sensor 100 may be the main operators of each step, in which case they may be referred to as a signal processing unit as an integrated functional block.

Here, an example of generating an integrated conversion function from N conversion functions will be described. In step S102, the image processing unit 511 calculates the overall average luminance and local average luminance of the image according to the luminance distribution of the scene. In step S104, the image processing unit 511 acquires a luminance histogram. The luminance histogram may be a logarithmic histogram.

The image processing unit 511 may obtain a luminance histogram based on the exposure conditions set for each pixel block 131. In this way, by using the luminance histogram generated to set the exposure conditions for each block 131, it is possible to omit the logarithmic conversion process of luminance.

In step S106, the image processing unit 511 determines a reference luminance and a band for each of the N conversion functions. The N reference luminances are different from one another, and are, for example, the overall average luminance calculated from a luminance histogram, an average luminance higher than the overall average luminance, and an average luminance lower than the overall average luminance. The image processing unit 511 may determine each band so that the median value of the reference luminance becomes the boundary of the band.

In step S108, the image processing unit 511 determines the combination ratio of the N transformation functions. The image processing unit 511 may determine the combination ratio according to the bandwidth of each transformation function.

The image processing unit 511 may determine the combination ratio of N conversion functions according to the luminance histogram. For example, the image processing unit 511 calculates a weighting coefficient according to the luminance frequency, and adjusts the combination ratio by multiplying the weighting coefficient by the combination ratio determined according to the bandwidth.

In step S110, the image processing unit 511 combines N conversion functions to generate an integrated conversion function. The equation for generating the integrated conversion function is as described in relation to Figure 7, so a description thereof will be omitted here.

In step S112, the arithmetic circuit 415 converts the luminance of the pixel signal using the integrated conversion function generated by the image processing unit 511.

Figure 9 is a diagram illustrating an example of steps for determining the reference luminance and band from a luminance histogram. Figure 9 corresponds to steps S104 to S108 of the luminance conversion flow described in Figure 8, and illustrates an example (N=3) of generating an integrated conversion function from three conversion functions 1 to 3.

9 shows the luminance histogram acquired in step S104. In step S106, the image processing unit 511 determines the overall average luminance, the average luminance of luminances lower than the overall average luminance, and the average luminance of luminances higher than the overall average luminance, and sets these as the reference luminances L _w,global,1 , L _w,global,2 , and L _w,global, 3 of the conversion functions 1 to 3, respectively.

Next, the image processing unit 511 determines an intermediate value Lw,med12 between the reference luminance Lw _,global,1 of conversion function 1 and the reference luminance Lw _,global,2 of conversion function 2, and an intermediate value Lw _{,med13 between} the reference luminance Lw _,global,1 of conversion function 1 and the reference luminance _Lw,global,3 of conversion function 3, and determines the range from the intermediate value Lw _,med12 to the intermediate value _Lw,med13 as band 1 of conversion function 1, the range from the minimum input luminance _Lw,min to the intermediate value _Lw,med12 as band 2 of conversion function 2, and the range from the intermediate value _Lw,med13 to the maximum input luminance _Lw,min as band 3 of conversion function 3.

In step S108, the image processing unit 511 determines the combination ratios k1 to k3 of transformation functions 1 to 3 based on the bandwidth ratios of bands 1 to 3.

Figures 10A and 10B show an example of a captured image whose luminance has been converted using a conversion function according to conventional technology. Figure 10C shows an example of a captured image whose luminance has been converted using the integrated conversion function according to this embodiment. Both images show the same outdoor scene captured from indoors through a window.

Figures 10A and 10B show images whose luminance has been converted using a conversion function that uses a single reference luminance as a reference. In Figure 10A, the reference luminance is set with emphasis on dark areas, resulting in blown-out highlights of pixels corresponding to outdoors. On the other hand, in Figure 10B, the reference luminance is set with emphasis on bright areas, resulting in crushed blacks of pixels corresponding to indoors.

Figure 10C shows an image whose luminance has been converted using an integrated conversion function that combines three conversion functions whose reference luminance is the overall average luminance, the average luminance of luminances lower than the overall average luminance, and the average luminance of luminances higher than the overall average luminance. As shown in Figure 10C, the luminance of bright areas is reduced and the luminance of dark areas is increased, thereby ensuring good reproduction of the gradation throughout the image without locally impairing the luminance of the actual scene.

Figure 11A shows an example of an image in which the luminance has been converted taking feature points into consideration. Figure 11A shows the image processing results of an image of a high-rise building photographed from indoors at night. Human vision recognizes the high-rise building as the center of the subject in a dark area. The image processing unit 511 extracts pixels corresponding to the high-rise building from the image as feature points, and determines the combination ratio of conversion functions according to the luminance of the feature points. The image processing unit 511 may include a conversion function that uses the luminance of the feature points as the reference luminance among the multiple conversion functions that form the basis of the integrated conversion function. Alternatively, the image processing unit 511 may generate an integrated conversion function that prioritizes the luminance of the feature points by multiplying the combination ratio corresponding to the band that includes the luminance of the feature points by a high weighting coefficient.

Figure 11B shows an example of a saliency map. A saliency map maps image features such as brightness, color, orientation, and direction, and is used to extract eye-catching features within an image. Figure 11B shows a saliency map corresponding to the scene in Figure 11A.

In Figure 11B, the areas corresponding to the illuminated windows and obstruction lights on the top floor of the high-rise building at the center of the subject, as well as the window frame near the photographer reflected in the image, stand out in bright colors, indicating that they are eye-catching features in the scene. In this way, by taking into account factors other than brightness, it is possible to reproduce the brightness of the scene naturally.

Figure 12 is a diagram illustrating another example of steps for determining the reference luminance and band from a luminance histogram. Like Figure 9, Figure 12 corresponds to steps S104 to S108 of the luminance conversion flow described in Figure 8. Here, explanations of parts common to Figure 9 will be omitted.

12, an integrated conversion function is generated from four conversion functions (N=4), including the same three conversion functions 1 to 3 as in FIG. 9, as well as conversion function 4, which uses peculiar luminance as a reference luminance. In step S106, the image processing unit 511 extracts peculiar luminance from the luminance histogram. The peculiar luminance is, for example, luminance that deviates from the luminance distribution but is detected with a significant frequency. The image processing unit 511 sets the extracted peculiar luminance as the reference luminance L _w,global,4 of conversion function 4. A description of how the band and combination ratio are determined will be omitted.

Next, the image processing unit 511 determines an intermediate value Lw,med12 between the reference luminance Lw _,global,1 of conversion function 1 and the reference luminance _Lw,global,2 of conversion function 2, and an intermediate value Lw _{,med13 between} the reference luminance Lw _,global,1 of conversion function 1 and the reference luminance Lw _,global,3 of conversion function 3, and determines the range from the intermediate value _Lw,med12 to the intermediate value _Lw,med13 as band 1 of conversion function 1, the range from the minimum input luminance _Lw,min to the intermediate value _Lw,med12 as band 2 of conversion function 2, and the range from the intermediate value _Lw,med13 to the maximum input luminance _Lw,min as band 3 of conversion function 3. The image processing unit 511 generates an integrated conversion function from four conversion functions, including a conversion function that uses anomalous luminance as the reference luminance in addition to multiple conversion functions that form the basis of the integrated conversion function.

In this way, even if a scene in the center of a dark area contains a point of significantly high brightness, such as an aircraft obstruction light on the top floor of a high-rise building as shown in Figure 11A, the brightness of the scene can be accurately reproduced by performing brightness conversion using an integrated conversion function that takes into account the unique brightness.

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," or the like, and it should be noted that processes can be performed in any order, unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that it is necessary to perform the processes in that order.

100 imaging element, 101 microlens, 102 color filter, 103 passivation film, 104 PD, 105 transistor, 106 PD layer, 107 wiring, 108 wiring layer, 109 bump, 110 TSV, 111 signal processing chip, 112 memory chip, 113 imaging chip, 131 block, 302 transfer transistor, 303 reset transistor, 304 amplification transistor, 305 selection transistor, 306 reset wiring, 307 TX wiring, 308 decoder wiring, 309 output wiring, 310 Vdd wiring, 311 load current source, 411 multiplexer, 412 signal processing circuit, 413 demultiplexer, 414 pixel memory, 415 arithmetic circuit, 420 drive control unit, 441 sensor control unit, 443 synchronization control unit, 444 Signal control unit, 460 setting unit, 500 imaging device, 501 system control unit, 503 photometry unit, 504 work memory, 505 recording unit, 506 display unit, 511 image processing unit, 512 calculation unit, 520 photographing lens

Claims (9)

a signal processing unit that generates an integrated conversion function by combining a plurality of conversion functions, the integrated conversion function including a first conversion function that converts an input luminance of a pixel signal into an output luminance, and a second conversion function that converts the input luminance into an output luminance different from the output luminance, and converts the luminance of the pixel signal using the integrated conversion function;
The first conversion function converts an input luminance into an output luminance using a first luminance as a reference luminance;
the second conversion function converts the input luminance into the output luminance using a second luminance different from the first luminance as a reference luminance;
The image processing device, wherein the signal processing unit determines a combination ratio of the plurality of conversion functions according to a bandwidth of a first band including the first luminance and a second band including the second luminance in a luminance distribution of the pixel signal. The image processing device according to claim 1 , wherein the signal processing unit determines a combination ratio of the plurality of conversion functions in accordance with a histogram of luminance of the pixel signal. The image processing device according to claim 1 , wherein the signal processing unit extracts feature points from the pixel signals and determines a combination ratio of the plurality of conversion functions according to the luminance of the feature points. The image processing device according to claim 1 , wherein the signal processing unit sets an average luminance calculated from a luminance histogram of the pixel signals as the first luminance. the plurality of conversion functions further include a third conversion function that converts an input luminance into an output luminance using a third luminance that is different from the first luminance and the second luminance as a reference luminance;
The image processing device according to claim 1 , wherein the signal processing unit sets an average luminance of pixel signals having a luminance higher than the first luminance to the second luminance, and sets an average luminance of pixel signals having a luminance lower than the first luminance to the third luminance. the signal processing unit extracts a peculiar luminance from a luminance distribution of the pixel signal;
The image processing device according to claim 1 , wherein the plurality of conversion functions further includes a conversion function that uses the anomalous luminance as a reference luminance. The image processing device according to claim 1 , wherein each of the transformation functions is an increasing function. The image processing device according to claim 7 , wherein each of the transformation functions is a sigmoid function. The image processing device according to claim 8 , wherein each conversion function converts input luminance into output luminance based on a reference luminance, an average luminance of pixel signals for each imaging region, and a predetermined brightness adjustment parameter.