JP7817824B2 - Image processing device, control method thereof, program, and storage medium - Google Patents

Description

The present invention relates to technology for correcting visible light images using invisible light images.

In the past, when photographing distant mountain ranges with a digital camera, the unevenness of the mountains could not be fully expressed due to the influence of haze such as dust in the air, resulting in a flat impression. For this reason, methods have been proposed to improve the contrast and resolution of images by using non-visible light images with longer wavelengths, such as infrared light, which is less affected by haze such as dust in the air.

For example, Patent Document 1 discloses a technology that uses an infrared light image to improve the perceived resolution of a visible light image. Specifically, the magnitude of the high-frequency components of the visible light luminance signal and the infrared light signal is determined, and the visible light luminance signal is corrected based on the results of that determination.

On the other hand, when photographing still lifes such as clothing or dolls, there is a desire to capture images that emphasize the texture of the fabric material. Even in such cases, invisible light such as infrared light has the characteristic that, unlike visible light, it is not affected by differences in spectral reflectance, even when there is a pattern, and the reflectance remains constant within the same material. Therefore, it is possible to emphasize only the texture of the fabric material.

JP 2016-082390 A

However, the conventional technology disclosed in the aforementioned Patent Document 1 performs correction only by comparing the intensity of the high-frequency components of the luminance signal, which is visible light, with the high-frequency components of the infrared light signal, and does not take into account spatial frequency or spatial distribution. As a result, it may not be the optimal process for emphasizing texture. Another issue is that there is no mechanism for controlling whether you want to emphasize texture or improve the contrast and resolution of the image depending on the shooting scene and your needs.

The present invention was made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can improve the quality of visible light images by using non-visible light images.

The image processing device of the present invention comprises an input means for inputting a visible light image and an invisible light image, a first generation means for generating an AC component of the visible light image, a second generation means for generating an AC component of the invisible light image, an adjustment means for adjusting the AC component of the invisible light image based on the AC component of the visible light image to obtain the adjusted AC component, and a correction means for correcting the visible light image using the adjusted AC component, wherein the adjustment means adjusts the AC component of the invisible light image depending on whether the image quality of the visible light image corrected by the correction means is required to prioritize image quality that gives priority to the texture of the subject, or image quality that gives priority to the contrast of the subject.

This invention makes it possible to improve the quality of visible light images by utilizing invisible light images.

1 is a block diagram showing the configuration of an imaging apparatus according to an embodiment of the present invention; 3A and 3B are diagrams illustrating a method for acquiring a visible light image and a non-visible light image. FIG. 2 is a block diagram showing the configuration of an image processing unit. 10 is a flowchart showing the flow of processing by an image processing unit. FIG. 2 is a block diagram showing the configuration of a visible light image correction processing unit. 10 is a flowchart showing the flow of processing by a visible light image correction processing unit. 3A and 3B are diagrams illustrating characteristics of image signals of a visible light image and a non-visible light image. 10A and 10B are diagrams illustrating a method for adjusting the invisible light AC component with priority given to texture and contrast. FIG. 4 is a block diagram showing the configuration of a non-visible light AC component adjustment unit. 10 is a flowchart showing the flow of processing by a non-visible light AC component adjustment unit. 10A and 10B are diagrams illustrating texture-priority and contrast-priority adjustments.

The following describes the embodiments in detail with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claimed invention. While the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the attached drawings, the same reference numbers are used to designate identical or similar components, and redundant explanations will be omitted.

Figure 1 is a block diagram showing the configuration of an imaging device 100, which is one embodiment of an image processing device of the present invention.

The imaging device 100 of this embodiment corrects visible light image signals using infrared light image signals, which have longer wavelengths than visible light, a type of invisible light. Two correction methods are available: a correction method that emphasizes texture, and a correction method that emphasizes resolution and contrast.

In FIG. 1, the imaging unit 101 includes a lens, an image sensor, an A/D conversion processing unit, and a development processing unit, and captures an image of a subject and generates an image file based on a control signal output from the system control unit 103 in response to an instruction from the operation unit 107.

In this embodiment, two types of signals are acquired: a visible light image and a non-visible light image. A possible signal acquisition method is shown in Figure 2. In Figure 2, the grid indicates the pixel arrangement of the image, with R, G, B, and W representing visible light signals and IR representing infrared light signals, which are non-visible light.

Explaining the acquisition methods in order from top to bottom, Figure 2(a) shows a method of acquiring visible light images and invisible light images using two image sensors, while Figure 2(b) shows a method of acquiring visible light images and invisible light images using a single image sensor. Figure 2(c) shows a method of acquiring visible light images acquired through an invisible light cut filter and invisible light images acquired through a visible light cut filter in time series. Visible light images and invisible light images can be acquired using the acquisition methods described above.

Returning to the explanation of Figure 1, the image processing unit 102 references the image files input from the imaging unit 101, recording unit 105, and network processing unit 106, and performs image processing to improve the image quality of the visible light image using the invisible light image based on the control signal output from the system control unit 103. The processing content of the image processing unit 102 related to this image processing will be explained in detail later.

The system control unit 103 controls and supervises the operation of the entire imaging device 100 by loading programs stored in the ROM 110 into the RAM 111 and executing them. For example, the system control unit 103 performs drive control of the imaging unit 101 based on instructions sent from the network processing unit 106 and the operation unit 107.

The display unit 104 receives the output signal from the image processing unit 102 and performs processing to display an image on a display device composed of a liquid crystal display or an organic EL (Electro Luminescence) display.

The recording unit 105 has the function of recording data such as images, and may include an information recording medium such as a memory card equipped with semiconductor memory or a package containing a rotating recording medium such as a magneto-optical disk, and this information recording medium may be removable.

The network processing unit 106 acquires image files from external input devices via a network. It also receives output signals from the image processing unit 102 and transmits the image signals via the network to an external display device or an external image processing device such as a PC (personal computer).

The bus 108 is used to send and receive data such as images between the imaging unit 101, image processing unit 102, system control unit 103, display unit 104, recording unit 105, network processing unit 106, ROM 110, and RAM 111.

Next, the configuration of the image processing unit 102 will be described. Figure 3 is a block diagram showing the configuration of the image processing unit 102 in this embodiment.

In this embodiment, the image processing unit 102 corrects the visible light image signal using the infrared light image signal, which has a longer wavelength than visible light as invisible light, as described above.

In FIG. 3, the image processing unit 102 is configured to include a visible light image input unit 201, a non-visible light image input unit 202, a visible light development processing unit 203, a non-visible light development processing unit 204, a visible light image correction processing unit 205, and an image output unit 206. The images input to the image processing unit 102 are a visible light image and a non-visible light image, and the output image is a visible light image corrected using the non-visible light image.

Figure 4 is a flowchart showing the flow of processing performed using the image processing unit 102. The operation of the flowchart in Figure 4 is realized by the system control unit 103 loading a program stored in ROM 110 into RAM 111 and executing it.

In step S501, the system control unit 103 performs processing to input a visible light image to the visible light image input unit 201 of the image processing unit 102.

In step S502, the system control unit 103 performs processing to input a non-visible light image to the non-visible light image input unit 202 of the image processing unit 102.

In step S503, the system control unit 103 uses the visible light development processing unit 203 of the image processing unit 102 to perform development processing on the visible light image input in step S501. Development processing for visible light images refers to general processing such as noise reduction processing, white balance processing, demosaic processing, color matrix processing, and gamma processing. In this embodiment, the visible light image signal after development processing is processed to convert the visible light R, G, and B signals into a luminance signal Y and a color difference signal UV, as shown in (Equation 1).

Y=0.299R+0.587G+0.114B
U=-0.169R-0.331G+0.500B...(Formula 1)
V=0.500R-0.419G-0.081B
In step S504, the system control unit 103 performs development processing on the non-visible light image input in step S502 using the non-visible light development processing unit 204 of the image processing unit 102. Development processing for non-visible light images refers to general processing such as noise reduction processing, demosaic processing, and gamma processing.

In step S505, the system control unit 103 uses the visible light image correction processing unit 205 of the image processing unit 102 to perform correction processing on the visible light image developed in step S503, using the non-visible light image developed in step S504. The processing of step S505 will be described in detail later.

In step S506, the system control unit 103 uses the image output unit 206 of the image processing unit 102 to perform processing to output the visible light image corrected in step S505.

The above is the operation of the image processing unit in this embodiment.

Next, Figure 5 is a block diagram showing the configuration of the visible light image correction processing unit 205. As described in step S505, the visible light image correction processing unit 205 performs correction processing on the visible light image using the non-visible light image.

In Figure 5, the visible light image correction processing unit 205 is configured to include a visible light AC component generation unit 301, an invisible light AC component generation unit 302, an invisible light AC component adjustment unit 303, and an invisible light AC addition unit 304. The images input to the visible light image correction processing unit 205 are the visible light image and the invisible light image developed in the previous processing, and the output image is a visible light image corrected using the invisible light image.

Figure 6 is a flowchart showing the processing flow of the visible light image correction processing unit 205 corresponding to step S505 in Figure 4. The operation of the flowchart in Figure 6 is realized by the system control unit 103 loading a program stored in ROM 110 into RAM 111 and executing it.

In step S601, the system control unit 103 uses the visible light AC component generation unit 301 to perform processing to generate AC components of the input visible light image. In this embodiment, the AC components of the visible light image are calculated from the luminance signal Y of the visible light image. The AC components are generated by applying a band-pass filter to the image, which extracts a specified frequency band, similar to general edge extraction processing.

In step S602, the system control unit 103 uses the non-visible light AC component generation unit 302 to generate AC components of the input non-visible light image. The generation of AC components is performed in the same manner as in step S601. Note that the characteristics of the band-pass filters used in steps S601 and S602 may be changed according to the spatial frequency information of the input image. For example, if the pixel size of the sensor that captures the non-visible light image is significantly larger than the pixel size of the sensor that captures the visible light image, the band-pass filter characteristics for the visible light image are designed to extract lower frequencies than the band-pass filter characteristics for the non-visible light image. This makes it possible to extract the same frequency bands from the visible light image and the non-visible light image.

In step S603, the system control unit 103 uses the non-visible light AC component adjustment unit 303 to adjust the AC components of the non-visible light image generated in step S602 based on the AC components of the visible light image generated in step S601. This generates the AC components of the non-visible light image to be used for correction. The method of adjusting the AC components of the non-visible light image will be explained in detail later.

In step S604, the system control unit 103 corrects the visible light image by using the non-visible light AC adder 304 to add the AC components of the non-visible light image adjusted in step S603 to the visible light image.

If the luminance signal of the corrected visible light image is Yout, the luminance signal of the visible light image before correction is Yin, and the AC component of the adjusted invisible light image is IR_ACadj, the luminance signal of the visible light image at coordinates (x, y) is corrected using (Equation 2).

Yout(x,y)=IR_ACadj(x,y)+Yin(x,y)...(Formula 2)
Note that (Equation 2) is the result of processing only AC components of a specific frequency, but it is also possible to process AC components of multiple frequencies in steps S601 to S603 and correct the luminance signal of the visible light image using (Equation 3), where i is a number used to distinguish the frequencies of the AC components.

Yout(x,y)=ΣiIR_ACadji(x,y)+Yin(x,y)...(Formula 3)
The above is the processing of the visible light image correction processing unit 205 in this embodiment.

Next, we will explain the processing of the non-visible light AC component adjustment unit 303, which corresponds to the processing in step S603. The non-visible light AC component adjustment unit 303 adjusts the AC components of the non-visible light image using both texture-priority and contrast-priority methods while referring to the AC components of the visible light image, and then combines the adjusted AC components to output the final AC components of the non-visible light image.

First, we will explain the gist of the processing within the non-visible light AC component adjustment unit 303 with reference to Figure 7.

Figure 7(a) is an image diagram of a vertically striped garment as the main subject, captured in a visible light image and an invisible light image. As shown in Figure 7(a), in the visible light image of the garment, the difference in color of the vertical stripes is expressed as pattern 901, whereas in the invisible light image of the garment, if the garment is made of the same material, the difference in color is not affected and only the mesh of the fabric of the garment is expressed.

The reason for this characteristic is that, as shown in Figure 7(b), the spectral reflectance range 903 that includes the color differences of the vertical stripes is included in the wavelength band of the spectral sensitivity characteristics 904 of the visible light sensor, but is not included in the wavelength band of the spectral sensitivity characteristics 905 of the invisible light sensor. Due to this characteristic, when the signal values of the vertical stripes are acquired horizontally, for example as in Figure 7(c), the signal values 906 of the visible light image clearly show the difference in signal values due to the difference in color of the vertical stripes. On the other hand, the signal values 907 of the invisible light image are not affected by the pattern itself, and only show the difference in signal values due to the mesh of the clothing fabric.

In this embodiment, this feature is used to adjust the invisible light AC component in a texture-priority mode that emphasizes the mesh of clothing fabric, and a contrast-priority mode that simply captures a high-contrast image.

Figure 8 shows the difference between texture-priority and contrast-priority adjustment methods. As in Figure 8(a), when you want to emphasize texture, it is desirable to use many areas where the distribution of the AC components of invisible light is uniform, regardless of the contours of the subject. Conversely, as in Figure 8(b), when you want to emphasize contrast by utilizing high-contrast areas of the invisible light image, it is desirable to use many areas where the amplitude of the AC components of invisible light is larger than the amplitude of the AC components of visible light.

As described above, the non-visible light AC component adjustment unit 303 realizes a configuration that allows adjustment modes based on two concepts to be used appropriately depending on the user's settings, the frequency to be adjusted, and the subject (depending on the conditions required for the image quality of the corrected visible light image).

Figure 9 is a block diagram showing the configuration of the non-visible light AC component adjustment unit 303. In Figure 9, the non-visible light AC component adjustment unit 303 is configured with a texture priority adjustment unit 401, a contrast priority adjustment unit 402, and a non-visible light AC synthesis unit 403. The input image is the AC component of a visible light image and the AC component of a non-visible light image, and the output image is the AC component of the adjusted non-visible light image (adjusted AC component).

Figure 10 is a flowchart showing the processing flow of the non-visible light AC component adjustment unit 303 corresponding to step S603 in Figure 6. The operation of the flowchart in Figure 10 is realized by the system control unit 103 loading a program stored in ROM 110 into RAM 111 and executing it.

In step S701, the system control unit 103 performs processing to adjust the AC components of the invisible light image, prioritizing texture, using the texture-priority adjustment unit 401 of the invisible light AC component adjustment unit 303. As mentioned above, this adjustment desirably employs many areas where the distribution of the invisible light AC components is uniform, regardless of the contours of the subject. Therefore, in this embodiment, the contours of the subject are identified based on the magnitude of the amplitude of the visible light AC components and the magnitude of the amplitude of the invisible light AC components, and adjustment is performed to employ only areas that are not contour parts and where the invisible light AC components are uniformly distributed.

In this embodiment, as shown in Figure 11(a), the adjustment shown in (Equation 4) is performed using an adjustment coefficient J based on the degree of contour and an adjustment coefficient K based on the degree of uniformity. IR_AC is the AC component of the invisible light image before adjustment, and IR_ACradj is the AC component of the invisible light image adjusted with texture priority. (x, y) indicates the position of interest in the image.

IR_ACradj(x,y)=J(x,y)×K(x,y)×IR_AC(x,y)…(Formula 4)
11A is calculated using Equation 5, and the uniformity U is calculated using Equation 6. IR_AC indicates the AC components of the invisible light image before adjustment, Y_AC indicates the AC components of the visible light image, i and j indicate the coordinates of the surroundings around the target position (x, y), and m and n indicate the surrounding range to be referenced.

E(x,y)=|IR_AC(x,y)|×|Y_AC(x,y)| …(Formula 5)

As can be seen from Equation 5 above, the contour degree E increases as the absolute values of both the AC components of the invisible light image and the visible light image increase, and the uniformity degree U increases as the standard deviation of the reference range of the AC components of the invisible light image decreases. In this embodiment, texture-prioritized adjustment is performed using the above method, but this is not limited to this, and adjustment may be performed using only either the contour degree adjustment coefficient J or the uniformity adjustment coefficient K.

Returning to the explanation of Figure 10, in step S702, the system control unit 103 performs processing to adjust the AC components of the invisible light image, prioritizing contrast, using the contrast priority adjustment unit 402 of the invisible light AC component adjustment unit 303. In this adjustment, as described above, it is desirable to use as many areas where the amplitude of the AC components of invisible light is larger than the amplitude of the AC components of visible light. Therefore, adjustment is performed to use only areas where the amplitude of the AC components of invisible light is larger than the amplitude of the AC components of visible light.

In this embodiment, as shown in Figure 11(b), adjustment is performed using (Equation 7) with an adjustment coefficient L that depends on the degree of contrast improvement. IR_AC is the AC component of the invisible light image before adjustment, and IR_ACcadj is the AC component of the invisible light image adjusted with contrast priority. (x, y) indicates the focus position on the image.

IR_ACcadj(x,y)=L(x,y)×IR_AC(x,y)...(Formula 7)
In this embodiment, the contrast improvement degree C on the horizontal axis of Fig. 11(b) is calculated using Equation 8. IR_AC is the AC component of the invisible light image before adjustment, Y_AC is the AC component of the visible light image, and (x, y) is the position of interest in the image.

C(x,y)=|IR_AC(x,y)|/(|Y_AC(x,y)|+1)...(Formula 8)
As can be seen from the above (Equation 8), the greater the ratio of the absolute value of the AC components of the invisible light image to the absolute value of the AC components of the visible light image, the greater the value of the contrast improvement degree C. In this embodiment, contrast-priority adjustment is performed using the above method, but the present invention is not limited to this. Adjustment may also be performed by comparing the magnitude of the amplitude of the AC components of the visible light image with the magnitude of the amplitude of the AC components of the invisible light image, and only when the amplitude of the AC components of the invisible light image is the magnitude of the amplitude of the AC components of the invisible light image greater, so that the AC components of the invisible light image are adopted.

In step S703, the system control unit 103 uses the invisible light AC synthesis unit 402 of the invisible light AC component adjustment unit 303 to synthesize the AC components of the invisible light image adjusted with texture priority in step S701 and the AC components of the invisible light image adjusted with contrast priority in step S702 (the two intermediate AC components). Then, the system control unit 103 generates the AC components of the finally adjusted invisible light image.

Let IR_ACadj be the AC component of the final adjusted invisible light image. IR_ACadj is calculated using a weighted average (Equation 9). IR_ACradj is the AC component of invisible light adjusted with priority on texture, and IR_ACcadj is the AC component of invisible light adjusted with priority on contrast. α is a blending coefficient that ranges from 0.0 to 1.0, and (x, y) indicates the focus position in the image.

IR_ACadj(x,y) =
α(x,y)×IR_ACradj(x,y)+(1.0−α(x,y))×IR_AC_cadj(x,y)
...(Formula 9)
In this embodiment, the blending coefficient α is a predetermined value that does not depend on the coordinates, but it may be a value determined based on the scene recognition results or user settings. Furthermore, the blending coefficient may be changed according to the coordinates (partial areas within the image) using subject recognition technology or the like. Furthermore, as described above, when adjusting multiple AC components with different frequencies, the value of the blending coefficient may be changed between a frequency band where texture enhancement is desired and a frequency band where it is not desired.

The above is the processing performed by the non-visible light AC component adjustment unit 303 in this embodiment.

As explained above, according to this embodiment, by using the non-visible light image in the optimal form depending on whether you want to emphasize texture or improve contrast or resolution, it is possible to appropriately correct the visible light image.

(Other embodiments)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

101: Image capture unit, 102: Image processing unit, 103: System control unit, 104: Display unit, 105: Recording unit, 106: Network processing unit, 107: Operation unit, 108: Bus, 110: ROM, 111: RAM

Claims (14)

an input means for inputting a visible light image and a non-visible light image;
a first generating means for generating an AC component of the visible light image;
a second generating means for generating an AC component of the non-visible light image;
an adjustment means for adjusting the AC components of the non-visible light image based on the AC components of the visible light image and acquiring the adjusted AC components;
a correction unit that corrects the visible light image using the adjusted AC component,
The image processing device is characterized in that the adjustment means adjusts the AC components of the non-visible light image depending on whether the image quality of the visible light image corrected by the correction means is required to prioritize the texture of the subject or the contrast of the subject . 2. The image processing device according to claim 1, wherein the adjustment means adjusts the AC components of the non-visible light image to generate a plurality of intermediate AC components, and combines the plurality of intermediate AC components in accordance with conditions required for the image quality of the visible light image to obtain the adjusted AC components. 3. The image processing apparatus according to claim 2 , wherein the adjustment means combines the plurality of intermediate AC components by taking a weighted average in accordance with conditions required for the image quality of the visible light image. 4. The image processing apparatus according to claim 3 , wherein said adjusting means changes a coefficient of the weighted average in accordance with a partial area within the image. 5. The image processing apparatus according to claim 2 , wherein the adjustment means generates two intermediate AC components as the plurality of intermediate AC components. 6. The image processing apparatus according to claim 5 , wherein the two intermediate AC components are a first intermediate AC component that prioritizes the texture of the object, and a second intermediate AC component that prioritizes the contrast of the object. 7. The image processing device according to claim 6 , wherein the adjustment means adjusts the AC component of the non-visible light image based on the amplitude of the AC component of the visible light image and the amplitude of the AC component of the non-visible light image. 8. The image processing device according to claim 7, wherein the first intermediate AC component is a portion of the AC component of the non-visible light image having a small amplitude, and the second intermediate AC component is a portion of the AC component of the non-visible light image having a larger amplitude than the first intermediate AC component. 9. The image processing device according to claim 1, wherein the correction means corrects the visible light image by adding the adjusted AC component to the visible light image. 10. The image processing device according to claim 1, wherein the non-visible light image is an image captured using infrared light, which has a longer wavelength than visible light. 11. The image processing device according to claim 1, further comprising an image capturing unit for capturing the visible light image and the non-visible light image. an input step of inputting a visible light image and a non-visible light image;
a first generation step of generating AC components of the visible light image;
a second generation step of generating AC components of the non-visible light image;
an adjusting step of adjusting the AC components of the non-visible light image based on the AC components of the visible light image to obtain adjusted AC components;
a correction step of correcting the visible light image using the adjusted AC components,
In the adjustment step, the AC components of the non-visible light image are adjusted depending on whether the image quality of the visible light image corrected in the correction step is required to prioritize the texture of the subject or the contrast of the subject . A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11 . 12. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.