JP7567542B2 - Image Processing Device - Google Patents

Description

This disclosure relates to an image processing device.

Conventionally, there has been known a technique for detecting the presence, posture, state, etc. of an object (e.g., a person) shown in a captured image obtained by capturing an image of a target area, such as the inside of a vehicle, with a capturing unit (e.g., a camera), and using the detection data for device control, etc. In such a case, the higher the resolution of the captured image, the better in order to achieve high accuracy in object detection.

In addition, for example, in order to ensure the resolution of the captured image and reduce the effects of ambient light, infrared lighting may be used both during the day and at night to capture images in the infrared region (infrared images). In such cases, for example, a monochrome image sensor that is sensitive to the infrared region may be used.

In addition, when considering displaying images on a display or performing image recognition using color information, it is desirable to obtain both RGB (Red, Green, Blue) images and infrared images. For example, when using a general color image sensor, each RGB pixel has sensitivity in the infrared region, so an RGB image can be obtained by using an infrared cut filter, and an infrared image can be obtained by using a visible light cut filter.

JP 2007-184805 A JP 2009-253447 A

Yuichi Nonaka et al., "Near-infrared color simultaneous imaging and noise reduction method based on visual characteristics," Journal of the Institute of Image Information and Television Engineers, Vol. 73, No. 1, 2019, pp. 177-189

However, when using a general color image sensor to obtain an infrared image, there is a problem that it is not possible to obtain a high-resolution infrared image because there is a difference in sensitivity in the infrared region for each RGB pixel color. In addition, there is also the problem that the sensitivity varies for each RGB pixel color due to the contamination of visible light components.

Therefore, one of the objectives of the present invention is to provide an image processing device that can obtain high-resolution infrared images by using images captured by a color image sensor.

An image processing device according to an embodiment of the present invention includes, for example, an acquisition unit that acquires a captured image having at least a first pixel having a first color component and an infrared component of visible light, a second pixel having a second color component and an infrared component of visible light, and a third pixel having a third color component and an infrared component of visible light from an imaging unit that receives light from a target area illuminated with infrared light by an irradiation unit through a visible light attenuation filter, a calculation unit that calculates an average value of luminance of a plurality of the first pixels, an average value of luminance of a plurality of the second pixels, and an average value of luminance of a plurality of the third pixels for a predetermined range in the captured image, a first correction gain value that is multiplied by the first pixel, and a second correction gain value that is multiplied by the second pixel. The device includes a determination unit that determines a gain value and a third correction gain value to be multiplied by the third pixel such that a ratio between the first correction gain value, the second correction gain value, and the third correction gain value is equal to a ratio between the reciprocal of the average luminance value of the first pixel, the reciprocal of the average luminance value of the second pixel, and the reciprocal of the average luminance value of the third pixel; and a generation unit that generates a monochrome image by using, for the specified range in the captured image, a value obtained by multiplying the luminance of each of the first pixels by the first correction gain value, a value obtained by multiplying the luminance of each of the second pixels by the second correction gain value, and a value obtained by multiplying the luminance of each of the third pixels by the third correction gain value.

According to this configuration, for example, when using an image captured by a color image sensor, by performing a brightness correction calculation using the above-mentioned first correction gain value, second correction gain value, and third correction gain value, it is possible to reduce the influence of the difference in sensitivity in the infrared region between the first pixel, the second pixel, and the third pixel, and to obtain a high-resolution monochrome image (infrared image).

In the image processing device described above, for example, the determination unit determines the first correction gain value, the second correction gain value, and the third correction gain value at a predetermined cycle or when a predetermined condition is satisfied. According to this configuration, for example, the first correction gain value, the second correction gain value, and the third correction gain value can be appropriately updated to continuously obtain high-resolution monochrome images (infrared images).

In the image processing device described above, for example, the calculation unit calculates an average luminance value of the first pixels, an average luminance value of the second pixels, and an average luminance value of the third pixels for the specified range in the captured image when the luminance difference between the first pixels is equal to or less than a specified value. With this configuration, for example, when the specified range is solid and the need for luminance correction of each pixel is high, the correction can be performed.

In the above-mentioned image processing device, for example, the first color component is green, the second color component is red, and the third color component is blue. With this configuration, for example, the above-mentioned correction can be performed when the three color components are the three primary colors of light.

FIG. 1 is a diagram showing an overall configuration of an image processing system according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing a Bayer type RGB pixel array according to the embodiment. FIG. 3 is a graph showing the relationship between the wavelength and sensitivity of an image signal according to the embodiment. FIG. 4 is a schematic diagram of a captured image and an enlarged image according to the embodiment. FIG. 5 is a flowchart showing the processing by the image processing circuit according to the embodiment. FIG. 6 is a schematic diagram showing a modified RGB-IR pixel array.

Below, an embodiment of an image processing system including an image processing device of the present disclosure will be described with reference to the drawings. The configuration of the embodiment described below, and the actions and effects brought about by said configuration, are merely examples, and the present invention is not limited to the contents described below. Furthermore, in the following embodiment, the image processing system is applied to a vehicle, as an example.

Figure 1 is an overall configuration diagram showing an overview of an image processing system S according to an embodiment. The image processing system S includes a camera 1, an ECU 6, and an infrared irradiation unit 7 (irradiation unit).

The ECU 6 controls various devices within the vehicle. The ECU 6 is composed of, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), etc. The CPU reads out programs stored in a non-volatile storage device such as a ROM, and executes various arithmetic processing according to the programs.

The RAM temporarily stores various data used in the calculations of the CPU. The SSD is a rewritable non-volatile storage unit, and can store data even when the power of the ECU 6 is turned off. Note that the ECU 6 may be configured to use other logical calculation processors such as a DSP (Digital Signal Processor) or logical circuits instead of a CPU. Also, a HDD (Hard Disk Drive) may be provided instead of the SSD, and the SSD and HDD may be provided separately from the ECU 6. Note that an ECU other than the ECU 6 may be provided inside the vehicle.

For example, the ECU 6 controls the camera 1 and the infrared irradiation unit 7 to capture images of the interior of the vehicle cabin, acquires captured images, detects the presence, posture, state, etc. of objects captured in the captured images, and controls various devices via the in-vehicle network 8 based on the detection data. The in-vehicle network 8 is configured, for example, as a CAN (Controller Area Network).

Therefore, it is desirable to place the camera 1 and the infrared irradiation unit 7 in a position that overlooks the interior of the vehicle. For example, the camera 1 and the infrared irradiation unit 7 are placed in the center of the upper inside part of the vehicle's windshield, but are not limited to this. The camera 1 and the infrared irradiation unit 7 may be configured as an integrated camera unit.

The infrared irradiating unit 7 is, for example, an LED (Light Emitting Diode) light that irradiates infrared rays. The irradiation angle and irradiation intensity of the infrared irradiating unit 7 are set so that the infrared light can be irradiated to an area that includes at least the area to be photographed by the camera 1.

The camera 1 mainly comprises a lens 2, a visible light attenuation filter 3, a color image sensor 4 (photographing section), and an image processing circuit 5. The lens 2 is a convex lens that serves as a light collecting means.

The visible light attenuation filter 3 is a filter that attenuates the visible light that passes through it. For example, the visible light attenuation filter 3 attenuates the visible light that passes through it by about 10%. By using such a visible light attenuation filter 3, it is possible to obtain the combined value of the infrared component and the reduced visible light component as the luminance (details will be described later). Note that the visible light attenuation filter 3 may be placed outside the lens 2 (above in FIG. 1).

The color image sensor 4 is a Bayer-type RGB image sensor. Normally, even RGB image sensors have sensitivity to IR (Infrared). For example, a typical color camera uses a Bayer-type RGB image sensor and is equipped with an IR cut filter in the optical system, detecting the single colors R, G, and B, not including IR, at each pixel. Figure 2 is a schematic diagram showing the pixel arrangement of a Bayer-type RGB sensor in an embodiment.

Next, FIG. 3 is a graph showing the relationship between the wavelength and sensitivity of an image signal in an embodiment. In FIG. 3, the horizontal axis is the wavelength of the image signal, and the vertical axis is the sensitivity of the image signal. Note that FIG. 3 is not a precise graph, but a rough graph.

In this embodiment, the visible light is attenuated by the visible light attenuation filter 3 to, for example, about 10%, and then received by the color image sensor 4. Lines L1, L2, and L3 respectively indicate the R output, G output, and B output, which have sensitivity differences near the boundary between visible light and infrared light. As a result, differences in luminance output for each pixel color (RGB) may occur. Also, when visible light is incident, differences in luminance for each pixel color (RGB) occur.

Here, FIG. 4 is a schematic diagram of a captured image and an enlarged image (hereinafter, both are also referred to as "images") in the embodiment. The image in FIG. 4 is a monochrome image that is converted using the brightness of each pixel based on the RGB output, and is the image before the correction described below is performed. Enlarged image B is an enlarged portion of captured image A. When focusing on RGB in enlarged image B, as described above, there is a sensitivity difference near the boundary between visible light and infrared light for the R output, G output, and B output, and also a brightness difference occurs when visible light is incident, so that an area that is actually plain turns into a checkerboard pattern. The correction to make this checkerboard pattern into a plain will be explained below.

Returning to FIG. 1, the image processing circuit 5 includes a processing unit 51 and a memory unit 52. The memory unit 52 stores various information and various calculation results required for the processing unit 51 to operate.

The processing unit 51 has, as its functional configuration, an acquisition unit 511, a calculation unit 512, a determination unit 513, a generation unit 514, and a control unit 515.

The acquisition unit 511 acquires various information from the color image sensor 4 and the ECU 6. For example, the acquisition unit 511 acquires a captured image from the color image sensor 4 that receives light (visible light, infrared light) from the capture target area irradiated with infrared light by the infrared irradiation unit 7 through the visible light attenuation filter 3. The captured image has at least a first pixel having a first color component and an infrared component of visible light, a second pixel having a second color component and an infrared component of visible light, and a third pixel having a third color component and an infrared component of visible light. For example, the first color component is green (G), the second color component is red (R), and the third color component is blue (B). Note that the captured image may have a fourth component (only IR), as will be described in a modified example below.

The calculation unit 512 calculates the average luminance value of a plurality of first pixels (G), the average luminance value of a plurality of second pixels (R), and the average luminance value of a plurality of third pixels (B) for a predetermined range in the captured image.

The specified range for which the brightness of the captured image is to be corrected may be the entire range of the captured image, each of the predetermined divided areas, or a range determined to be a plain area.

When the calculation unit 512 determines that a predetermined range in the captured image is, for example, a plain area, it calculates the average luminance value of a plurality of first pixels (G), the average luminance value of a plurality of second pixels (R), and the average luminance value of a plurality of third pixels (B).

Examples of blank areas include walls and ceilings in the area to be photographed. Therefore, for example, the average brightness and variance (histogram) within the calculation area (e.g., 6x6 pixels, 8x8 pixels, etc.) of the photographed image are calculated for each RGB color component, and areas with little change in brightness are determined to be blank areas. Note that only one type of pixel, such as only the G pixels (hereinafter simply referred to as "G") of RGB, may be used in the calculation. In other words, for example, an area may be determined to be blank when the brightness difference between multiple first pixels (G) is equal to or less than a predetermined value.

When only G is used in the calculation, the blank area is determined, for example, as follows.
First, the average brightness value of G (Gave) in the calculation target region is calculated.
Next, the maximum brightness value of G (Gmax) in the calculation target region is calculated.
Next, the minimum luminance value of G (Gmin) in the calculation target region is calculated.
The threshold value (Cg) is set to a value of about 10% to 20% of Gave.

If the condition "Gmax - Gmin < threshold (Cg)" is met, it is determined to be a blank area. In order to remove noise components, it is preferable to remove maximum and minimum values that are not within a specified range, or to calculate from the variance or deviation (σ).

A similar determination may also be made for R and B, and if the conditions are met for all RGB, it may be determined that the area is a blank area.

The determination unit 513 determines the first correction gain value to be multiplied by the first pixel (G), the second correction gain value to be multiplied by the second pixel (R), and the third correction gain value to be multiplied by the third pixel (B) such that the ratio of the first correction gain value, the second correction gain value, and the third correction gain value is equal to the ratio of the reciprocal of the average luminance value of the first pixel (G), the reciprocal of the average luminance value of the second pixel (R), and the reciprocal of the average luminance value of the third pixel (B).

For example, the first correction gain value is Gg, the second correction gain value is Gr, and the third correction gain value is Gb. Also, the average luminance value of the first pixel (G) is [G], the average luminance value of the second pixel (R) is [R], and the average luminance value of the third pixel (B) is [B].

In this case, for example, each correction gain value can be calculated using G as a reference as follows.
Gg=[G]/[G]=1
Gr=[G]/[R]
Gb=[G]/[B]

The determination unit 513 newly determines (updates) the first correction gain value, the second correction gain value, and the third correction gain value at a predetermined period or when a predetermined condition is satisfied. The reason for this update is as follows.

Although infrared light is used for shooting, the visible light is not completely blocked by the visible light attenuation filter 3. As the light environment changes from moment to moment, the visible light component is disturbance light, and the ratio of the amount of visible/infrared light received by the color image sensor 4 is not constant but can change. Therefore, the appropriate correction gain values can also change, so it is preferable to update each correction gain value as appropriate.

The specified cycle is, for example, every frame, every few seconds, or every few minutes. In addition, when a specified condition is satisfied, it is when a change in the light environment of the area to be photographed is detected, specifically, when any of (1) to (3) below occurs.

(1) Change in average luminance in the captured image (whole image, each correction area)
(2) Changes in camera exposure time (detect changes in light intensity)
(3) Changes in the brightness histogram (entire image, each correction area)

Note that, for example, under conditions where there is almost no visible light, such as at night, the correction gain values are unlikely to change significantly, so it may be possible not to update the correction gain values.

The generating unit 514 generates a monochrome image by using a value obtained by multiplying the luminance of each first pixel (G) by a first correction gain value, a value obtained by multiplying the luminance of each second pixel (R) by a second correction gain value, and a value obtained by multiplying the luminance of each third pixel (B) by a third correction gain value for a predetermined range in the captured image. This causes the average luminance values of RGB after correction to match, thereby suppressing the checkerboard pattern caused by the sensitivity difference between RGB.

In the above calculation example, each correction gain value is 1 or a value close to 1. Therefore, for example, a normal range for each correction gain value may be determined, and if the normal range is exceeded, a warning may be issued indicating that there is some kind of problem. The normal range may be determined, for example, from the maximum sensitivity difference between the colors of the color image sensor 4, the characteristics of the visible light attenuation filter 3, etc.

The control unit 515 performs processes and controls other than those performed by the units 511 to 514. For example, the control unit 515 stores the monochrome image generated by the generation unit 514 in the memory unit 52 and transmits it to the ECU 6. The control unit 515 also stores a color image based on the RGB output from the color image sensor 4 in the memory unit 52 and transmits it to the ECU 6.

Figure 5 is a flowchart showing the processing by the image processing circuit 5 of the embodiment. First, in a state where the infrared light irradiation unit 7 is irradiating the shooting target area with infrared light, in step S1, the acquisition unit 511 acquires a shot image from the color image sensor 4.

Next, in step S2, the control unit 515 performs black level correction on the captured image acquired in step S1.

Next, in step S3, the calculation unit 512 cuts out a processing area from the captured image.

Next, in step S4, the processing unit 51 determines whether it is time to correct the error. That is, the processing unit 51 determines that it is time to correct the error when the above-mentioned predetermined period or predetermined condition is satisfied.

Next, in step S5, the calculation unit 512 extracts a correction area in the captured image. For example, as described above, the range determined to be a blank area in the captured image is set as the correction area.

Next, in step S6, the calculation unit 512 calculates the average luminance (average luminance value) for each pixel color (RGB) for the correction area in the captured image.

Next, in step S7, the determination unit 513 determines the correction gain value for each pixel color (RGB).

Next, in step S8, the generation unit 514 multiplies the luminance by the correction gain value for each RGB pixel to generate a monochrome image.

In this way, according to the image processing system S of this embodiment, when using an image captured by the color image sensor 4, the above-mentioned first correction gain value Gg, second correction gain value Gr, and third correction gain value Gb are used to perform brightness correction calculations, thereby reducing the influence of the difference in sensitivity in the infrared region between the first pixel (G), second pixel (R), and third pixel (B), and a high-resolution monochrome image (infrared image) can be obtained. In other words, since the average brightness values of RGB after correction are the same, checkerboard patterns caused by sensitivity differences between RGB can be suppressed.

Therefore, a commonly available color image sensor 4 can be used to stably obtain infrared images with the same high resolution as a monochrome IR image sensor. This technology can then be applied to, for example, a DMS (Driver Monitoring System) etc.

For example, such high-resolution monochrome images can be used to detect the presence, posture, and state of an object (e.g., a person) captured within the target area, and the detection data can be used for device control, etc. Specifically, for example, when recognizing the line of sight of a vehicle occupant using image processing, Purkinje images and pupils can be detected with high accuracy.

In addition, by appropriately updating the first correction gain value Gg, the second correction gain value Gr, and the third correction gain value Gb, it is possible to continuously obtain high-resolution monochrome images (infrared images).

In addition, when a specified range is solid and there is a high need for brightness correction for each pixel, such correction can be performed.

The above correction can also be performed when the three color components are the three primary colors of light.

In addition, by using the visible light attenuation filter 3, it is possible to obtain both an infrared image and a color image (RGB image) without having to exchange a visible light blocking filter for an infrared blocking filter as in conventional technology; in other words, it is possible to obtain the combined value of the infrared component and the reduced visible light component as brightness. The color image can then be used for image recognition using color information and for display on a display, contributing to the addition of sensing and recognition functions and improved accuracy that were not possible with conventional configurations.

(Modification)
Modifications will be described below. In the above embodiment, the case of a Bayer type RGB pixel array as shown in FIG. 2 has been described. The present invention is not limited to this, and can also be applied to the case of an RGB-IR pixel array. In other words, an RGB-IR image sensor can be used as the color image sensor 4 instead of the RGB image sensor of the Bayer array. This RGB-IR image sensor does not have an IR cut filter, and is provided with pixels that are sensitive only to IR so that only the RGB components can be extracted from the detected signal.

Figure 6 is a schematic diagram showing a modified RGB-IR pixel array. Each RGB pixel generally has sensitivity in the IR region that is approximately the same as an IR-only pixel (fourth pixel). Therefore, the RGB output from an RGB-IR image sensor can be expressed as "R+IR", "G+IR", and "B+IR". Therefore, by subtracting the IR component from the RGB output, it is possible to extract each RGB color.

Furthermore, the time when the above-mentioned predetermined condition is satisfied may be, for example, the time (4) below.
(4) When the ratio of the average brightness of the IR pixel to the average brightness of each of the R, G, and B pixels becomes equal to or greater than a certain value (for the entire image or each correction area)

The program executed by the image processing circuit 5 of this embodiment may be configured to be provided by recording it in an installable or executable format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

Furthermore, the program may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program may also be configured to be provided or distributed via a network such as the Internet.

Although the embodiments and variations of the present invention have been described, these embodiments and variations are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and variations are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents as set forth in the claims.

For example, if separate correction gain values are calculated for two adjacent plain areas in a captured image, the correction gain value may be gradually changed at the boundary.

In addition, some of the functions of the image processing device may be realized by the ECU 6.

1...camera, 2...lens, 3...visible light attenuation filter, 4...color image sensor, 5...image processing circuit, 6...ECU, 7...infrared irradiation unit, 8...in-vehicle network, 51...processing unit, 52...storage unit, 511...acquisition unit, 512...calculation unit, 513...determination unit, 514...generation unit, 515...control unit, S...image processing system

Claims (4)

an acquisition unit that acquires a captured image having at least a first pixel having a first color component and an infrared component of visible light, a second pixel having a second color component and an infrared component of visible light, and a third pixel having a third color component and an infrared component of visible light from an imaging unit that receives light from a target area illuminated with infrared light by an illumination unit through a visible light attenuation filter;
a calculation unit that calculates an average value of luminance of a plurality of the first pixels, an average value of luminance of a plurality of the second pixels, and an average value of luminance of a plurality of the third pixels for a predetermined range in the captured image;
a determination unit that determines a first correction gain value by which the first pixel is multiplied, a second correction gain value by which the second pixel is multiplied, and a third correction gain value by which the third pixel is multiplied, such that a ratio of the first correction gain value, the second correction gain value, and the third correction gain value is equal to a ratio of a reciprocal of an average value of luminance of the first pixel, a reciprocal of an average value of luminance of the second pixel, and a reciprocal of an average value of luminance of the third pixel;
and a generation unit that generates a monochrome image by using, for the specified range in the captured image, a value obtained by multiplying a luminance of each of the first pixels by the first correction gain value, a value obtained by multiplying a luminance of each of the second pixels by the second correction gain value, and a value obtained by multiplying a luminance of each of the third pixels by the third correction gain value. The image processing device according to claim 1, wherein the determination unit newly determines the first correction gain value, the second correction gain value, and the third correction gain value at a predetermined interval or when a predetermined condition is satisfied. The image processing device according to claim 1, wherein the calculation unit calculates an average luminance value of the first pixels, an average luminance value of the second pixels, and an average luminance value of the third pixels for the predetermined range in the captured image when the luminance difference of the first pixels is equal to or less than a predetermined value. The image processing device of claim 1, wherein the first color component is green, the second color component is red, and the third color component is blue.

Images