CN110620885B - Infrared low-light-level image fusion system and method and electronic equipment - Google Patents

Abstract

The application relates to an infrared low-light-level image fusion system and method and electronic equipment. The method comprises the following steps: the infrared image acquisition module: the infrared image acquisition module is used for acquiring an infrared image; the low-light-level image acquisition module: used for collecting low-light level images; the video data conversion module: the multi-channel video decoding chip is used for decoding and converting the infrared image and the low-light-level image from analog signals into digital signals and outputting the digital signals to the FPGA processing module; an FPGA processing module: the system is used for outputting a fused color image after preprocessing, image registration and image fusion operations are carried out on the infrared image and the low-light-level image through the FPGA chip; an A/D conversion module: and the A/D conversion chip is used for converting the color image into a VGA analog signal and displaying the VGA analog signal through the display module. This application has designed the collection and the fusion system of infrared and shimmer image through infrared shimmer image acquisition device and FPGA data processing controlling means, and the fusion speed of image can improve greatly, and image stereovision is stronger, is favorable to the observation and the judgement to the image.

Description

An infrared low-light image fusion system, method and electronic device

technical field

The present application belongs to the technical field of digital image processing, and in particular relates to an infrared low-light image fusion system, method and electronic device.

Background technique

The infrared image is an image generated by the radiation of the object itself, which can "actively" obtain the target information in the scene, and can well display the hidden thermal target, which is less affected by the lighting conditions and bad weather, but at the same time due to the imaging Due to the limitation of the principle, the infrared image has low contrast, strong spatial correlation, poor ability to reflect the details of the target, and the imaging effect does not conform to the visual habits of the human eye. Since the image captured by the visible light sensor is the reflection image of the object, the formed image contains rich detailed information, and the imaging effect conforms to the observation habits of the human eye. Low-light images belong to the category of visible light, especially under low illumination, their content information is richer than that of infrared images, which can better describe the details of the environment, but limited by the environment and distance, the imaging noise is very high when the weather is bad. It is easy to lose the target, especially when the chromaticity difference between the target and the background is small.

Fusion of infrared image and low-light image can greatly facilitate the information acquisition of the two images by the detector, and can maintain their respective advantages. The fusion of infrared images and low-light images can effectively utilize their respective characteristic part information, enhance the ability to understand the scene, highlight the target, help to detect concealed and camouflaged targets, and can improve the ability of night monitoring and night combat. However, due to the differences in the imaging mechanism of infrared images and low-light images, the environment and time for obtaining images are different, and the resulting images will inevitably have certain differences in space and time, resulting in the fusion of images. There will be big differences. The current image registration method mainly focuses on software implementation, which takes a long time and has poor real-time performance.

Application No. 201710300515.7 discloses an optical image fusion system and method for infrared and night vision devices. The system includes a night vision device, and the beam splitter includes a first side surface and a second side surface opposite to each other, and is arranged obliquely upstream of the optical path of the night vision device; the long-wave infrared light emitted by the target is reflected by the first side surface, and the reflected light passes through a first light-converging device; the near-infrared light of the target enters the night vision device after passing through the first side and the second side in turn; the reflected light passes through the first light-converging device and is imaged on a photodetector; the photodetector The output electrical signal is sent to the processing unit; the processing unit converts the electrical signal into an image signal; the display displays a visible light image corresponding to the thermal imaging of the target according to the image signal; Reflected, reflected light enters the night vision device. This method is a pure optical system and does not involve image fusion. The multiple reflections of the incident light will cause a large loss, which will greatly interfere with image acquisition.

Application No. 201710423679.9 discloses a real-time fusion system of infrared images and low-illumination images. The system includes a low-illumination optical lens group, an infrared optical lens group, a synchronous focusing device, a low-illumination image sensor, an infrared image sensor, and a low-illumination image. Sensor driver module, infrared image sensor driver module, image sensor analog front end, low-light image processor, image synchronous acquisition module, digital image processing module, image display control module, microdisplay, display magnifying lens group, power supply module, processor peripheral circuit , display the switch button. The system adopts synchronous focusing, and performs image registration and fusion with the existing software algorithm. The efficiency of registration and fusion is not high, the real-time performance of the obtained fusion image is poor, and the output delay is high.

SUMMARY OF THE INVENTION

The present application provides an infrared low-light image fusion system, method and electronic device, aiming to solve one of the above-mentioned technical problems in the prior art at least to a certain extent.

In order to solve the above problems, the application provides the following technical solutions:

An infrared low-light image fusion system, comprising:

Infrared image acquisition module: used to acquire infrared images;

Low-light image acquisition module: used to collect low-light images;

Video data conversion module: used to decode and convert the infrared image and low-light image from analog signal into digital signal through multi-channel video decoding chip, and after frame synchronization and frame buffer processing of the two digital signals, output to FPGA for processing module;

FPGA processing module: used to output the fused color image after preprocessing, image registration and image fusion operations on the infrared image and the low-light image through the FPGA chip;

A/D conversion module: used to convert the color image into a VGA analog signal using an A/D conversion chip, and display it through the display module.

The technical solutions adopted in the embodiments of the present application further include: the FPGA processing module specifically includes:

Image preprocessing unit: used for filtering the infrared image and the low-light image;

Image registration unit: used to perform optical registration, image cropping and image scaling operations on the filtered infrared image and low-light image;

Image fusion unit: It is used to fuse the registered infrared image and low-light image by using the MIT pseudo-color image fusion algorithm, and output the fused color image.

The technical solutions adopted in the embodiments of the present application further include: the image registration unit performs optical registration, image clipping and image scaling operations on the infrared image and the low-light image, specifically including:

The dual-channel parallel optical axis system is used for image registration; it is assumed that the elevation angles of the light that the infrared lens and the low-light lens can receive in the horizontal and vertical directions are the same, and the overlap of the collected infrared image and low-light image is calculated accordingly. The size and area occupied by the section:

In the above formula, u is the distance between the target and the objective lens, b is the distance between the two objective lenses in the parallel optical axis, and θ is the horizontal field of view of the horizontal lens;

Cut the non-common parts in the infrared image and the low-light image, first extract the data, count the input data, including the number of rows and columns, determine the cutting range of the input image, and determine whether each input data is being cut. When the data is within the cutting range, the data is input into the frame buffer, and when the data is not within the cutting range, the data is not processed, and the judgment of the next data is started;

When the rest of the two images are common parts, scaling processing is performed on the two images, and the image scaling includes a data buffer unit, a bilinear interpolation operation unit, and a coefficient generation and logic control unit; the data buffer unit Two FIFOs are used for line buffering of data, and ping-pong operation is used for data line buffer writing; when data interpolation is performed, the logic control unit controls the data read-in and write-out of the data buffer unit, and generates interpolation parameters at the same time , input into the bilinear interpolation operation unit, the bilinear interpolation operation unit performs interpolation in the Y direction for the two input line buffer data lines, and after two interpolations in the Y direction, the interpolation in the Y direction can be obtained. The two results are then interpolated in the X direction, and the interpolated data can be obtained by performing the interpolation in the X direction once.

The technical solution adopted in the embodiment of the present application further includes: the image fusion unit uses the MIT pseudo-color image fusion algorithm to perform fusion processing on the two registered images. Specifically: through 6 convolvers, 4 dividers, 4 A normalization process and a delay output unit implement the MIT fusion algorithm; images are extracted from the infrared image frame buffer and the low-light image frame buffer by means of synchronous extraction, and the extracted images are entered into the corresponding convolvers respectively. Perform convolution, the convolution result is input into the divider, and finally normalized by the normalization process, so that the range of the result is between 0 and 256, so that the image is enhanced; then input the infrared image and the low-light image Perform image fusion, and finally map the fusion results to the three RGB channels to obtain a fused color image.

The technical solutions adopted in the embodiments of the present application also include:

External memory module: including the memory chip SDRAM chip used to cache the real-time image and exchange real-time data with the FPGA chip in the image processing process, and the memory chip FLASH chip used to store the FPGA configuration information;

External control module: used to adjust the internal mode of the FPGA chip through the button, and control the output of different mode signals.

Another technical solution adopted in the embodiment of the present application is: a method for fusion of infrared low-light images, comprising the following steps:

Step a: collecting infrared images and low-light images respectively;

Step b: decode and convert the infrared image and the low-light image from the analog signal into a digital signal by using a multi-channel video decoding chip, and perform frame synchronization and frame buffer processing on the two-channel digital signal, and then output to the FPGA processing module;

Step c: After preprocessing, image registration and image fusion operations are performed on the infrared image and the low-light image through the FPGA chip, the fused color image is output;

Step d: use an A/D conversion chip to convert the color image into a VGA analog signal, and display it through a display module.

The technical solutions adopted in the embodiments of the present application further include: in the step c, the preprocessing, image registration, and image fusion operations performed on the infrared image and the low-light image by the FPGA chip specifically include:

Step c1: filter the infrared image and the low-light image;

Step c2: performing optical registration, image cropping and image scaling operations on the filtered infrared image and the low-light image;

Step c3: Image fusion unit: used to fuse the registered infrared image and the low-light image by using the MIT pseudo-color image fusion algorithm, and output the fused color image.

The technical solution adopted in the embodiment of the present application further includes: in the step c2, the performing optical registration, image cropping and image scaling operations on the infrared image and the low-light image specifically includes: using a dual-channel parallel optical axis system Perform image registration; assuming that the elevation angles of the light that the infrared lens and the low-light lens can receive in the horizontal and vertical directions are the same, calculate the size and area of the overlapped part of the collected infrared image and low-light image:

In the above formula, u is the distance between the target and the objective lens, b is the distance between the two objective lenses in the parallel optical axis, and θ is the horizontal field of view of the horizontal lens;

Cut the non-common parts in the infrared image and the low-light image, first extract the data, count the input data, including the number of rows and columns, determine the cutting range of the input image, and determine whether each input data is being cut. When the data is within the cutting range, the data is input into the frame buffer, and when the data is not within the cutting range, the data is not processed, and the judgment of the next data is started;

When the rest of the two images are common parts, scaling processing is performed on the two images, and the image scaling includes a data buffer unit, a bilinear interpolation operation unit, and a coefficient generation and logic control unit; the data buffer unit Two FIFOs are used for line buffering of data, and ping-pong operation is used for data line buffer writing; when data interpolation is performed, the logic control unit controls the data read-in and write-out of the data buffer unit, and generates interpolation parameters at the same time , input into the bilinear interpolation operation unit, the bilinear interpolation operation unit performs interpolation in the Y direction for the two input line buffer data lines, and after two interpolations in the Y direction, the interpolation in the Y direction can be obtained. The two results are then interpolated in the X direction, and the interpolated data can be obtained by performing the interpolation in the X direction once.

The technical solution adopted in the embodiment of the present application further includes: in the step c3, the fusion processing of the two registered images by using the MIT pseudo-color image fusion algorithm is specifically: using 6 convolvers, 4 A divider, 4 normalization processes and a delay output unit implement the MIT fusion algorithm; images are extracted from the infrared image frame buffer and the low-light image frame buffer by means of synchronous extraction, and the extracted images enter the corresponding The convolution is performed in the convolver, the convolution result is input into the divider, and finally normalized by the normalization process, so that the range of the result is between 0 and 256, so that the image is enhanced; then input the infrared image Perform image fusion with the low-light image, and finally map the fusion results to the three RGB channels to obtain the fused color image.

Another technical solution adopted in the embodiment of the present application is: an electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor, so that the at least one processor can perform the following operations of the above-mentioned infrared low-light image fusion method:

Step a: collecting infrared images and low-light images respectively;

Step b: decode and convert the infrared image and the low-light image from the analog signal into a digital signal by using a multi-channel video decoding chip, and perform frame synchronization and frame buffer processing on the two-channel digital signal, and then output to the FPGA processing module;

Step c: After preprocessing, image registration and image fusion operations are performed on the infrared image and the low-light image through the FPGA chip, the fused color image is output;

Step d: use an A/D conversion chip to convert the color image into a VGA analog signal, and display it through a display module.

Compared with the prior art, the beneficial effects of the embodiments of the present application are: the infrared low-light image fusion system, method and electronic device of the embodiments of the present application design infrared and micro-light images through the infrared low-light image acquisition device and the FPGA data processing control device. Compared with the prior art, the optical image collection and fusion system has at least the following advantages:

1. In the image registration part, the image registration method is mainly used for parallel optical axis image registration, supplemented by digital image registration. When performing digital image registration, only the image needs to be cut and zoomed, which can be extremely The speed of image processing is greatly improved, and the real-time performance of image processing will be greatly improved;

2. The image fusion part uses FPGA to implement MIT's pseudo-color image fusion algorithm. Due to the characteristics of FPGA's operation parallelism, the image fusion speed will be greatly improved, the image output delay is very low, and the real-time performance is very strong. Secondly, the MIT algorithm fusion The latter is a color image, and the image has a stronger sense of hierarchy, which is more conducive to the observation and judgment of the image;

3. The entire image fusion system uses FPGA as the core chip for signal control and operation, and the circuit system is simple to build without the assistance of other processors, which is easy to implement.

Description of drawings

Fig. 1 is the hardware structure diagram of the infrared low-light image fusion system of the embodiment of the present application;

2 is a schematic structural diagram of an infrared low-light image fusion system according to an embodiment of the present application;

Fig. 3 is the optical path diagram of the parallel optical axis system;

Fig. 4 is a flow chart of image cutting;

5 is a schematic diagram of an image scaling process;

6 is a schematic diagram of an image fusion algorithm framework;

7 is a flowchart of an infrared low-light image fusion method according to an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a hardware device of the infrared low-light image fusion method provided by the embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

Please refer to FIG. 1 , which is a hardware structure diagram of an infrared low-light image fusion system according to an embodiment of the present application. The infrared low-light image fusion system of the embodiment of the present application uses the XC6SLX45-2FG484C chip of the SPARTAN6 series of Xilinx Company, and the development software platform is ISE Design Suite 14.7 software. The whole hardware system consists of power supply circuit, frame buffer circuit, video data format conversion circuit, FPGA main processor circuit, D/A digital-to-analog conversion circuit, etc., including SDRAM module, SPIFLASH module, JTAG module and crystal oscillator module, and takes FPGA as the The core processor, driven by the power module, receives the input video data, and then converts it into analog signal output after being processed by the FPGA processor.

The system image processing mechanism is mainly composed of FPGA+DDR3. The FPGA is a high-speed FPGA chip of XC6SLX45-2FG484 of XILINX's SPARTAN6 series, which is responsible for the core algorithm of video image processing, making full use of the parallel processing capability of FPGA, plus the connection between FPGA and DDR3. High-speed data reading and writing, the bandwidth of the entire system is up to 10Gb/s (666M*16bit), and the DDR3 capacity is up to 2Gbit, which meets the demand for high buffers in the video processing process. The clock frequency of the communication between the hard Core and DDR3 integrated with the DDR controller in the FPGA reaches 333Mhz, and the internal DDR3 is 666Mhz. The two-channel video input acquisition of infrared image and low-light image adopts Techwell's TW2867, which can input up to 4 channels of composite video signals, PAL/NTSC/SECAM automatic recognition, output BT656, multiplex bus, and FPGA Multiplexing, saving IO. The video output adopts ADI's three-channel, 10-bit DAC conversion chip ADV7123, which supports RGB digital input and VGA interface output. It supports the conversion rate of up to 240MSPS, and supports up to 1080p 60Hz video image output.

Please refer to FIG. 2 , which is a schematic structural diagram of an infrared low-light image fusion system according to an embodiment of the present application. The infrared low-light image fusion system of the embodiment of the present application includes an infrared image acquisition module, a low-light image acquisition module, a video data conversion module, an FPGA processing module, an A/D conversion module, a display module, an external storage module, and an external control module.

Infrared image acquisition module: used to collect infrared images; among them, the infrared image acquisition module of the present application adopts the UWA384CX-H42 model infrared sensor core and is equipped with an infrared lens, and the lens surface has a filter film, which reflects visible light but allows The light of infrared wavelengths passes through, thereby reducing the passage of visible light and increasing the transmittance of infrared light. The lens has a light-gathering effect, which can condense the infrared light radiated or reflected from the outside through the lens and then project it on the back IRFPA array (infrared focal plane array). ), the IRFPA array senses infrared light and images it.

Low-light image acquisition module: used to collect low-light images; wherein, the low-light image acquisition module of the present application adopts 1XC18/18WHS-CL model image intensifier core, and adopts an adjustable aperture optical lens, and its output is PAL video Mode, the external light information is concentrated in the photosensitive array after passing through the optical lens, and the image intensifier enhances the image on the convergence and focal plane to increase the light sensitivity of the system and enhance low-light imaging.

Video data conversion module: It is used to decode and convert the collected infrared images and low-light images from analog signals into digital signals through a multi-channel video decoding chip, and after frame synchronization and frame buffer processing of the two digital signals, output to FPGA for processing module; in the embodiment of this application, since the video signals output by image acquisition devices such as infrared optical lenses, visible light optical lenses, infrared image detectors, and low-light image detectors are all in PAL (Phase Alteration Line, Pal system) mode, the follow-up The FPGA chip cannot be directly processed, so it is necessary to use a video decoding chip to convert infrared images and low-light images from video analog signals to digital signals, and then send them to the FPGA for processing.

FPGA processing module: used to perform related preprocessing, image registration and image fusion operations on infrared images and low-light images to obtain a fused color image; specifically, the FPGA processing module includes:

Image preprocessing unit: used to filter infrared images and low-light images;

Image registration unit: used to perform optical registration, image cropping and image scaling operations on the filtered infrared image and low-light image; wherein, the application uses a dual-channel parallel optical axis system to perform the registration of the front-end optical system , which will not interfere with each other, which is conducive to the further fusion of images. The following table 1 is the relevant parameters of the infrared lens and the low-light lens used in this application:

Table 1 Parameters of infrared lens and low-light lens

As shown in Figure 3, it is the optical path diagram of the parallel optical axis system. As shown in Table 1, since the horizontal and vertical field angles of the infrared lens and the low-light lens are not much different, it can be approximately considered that the elevation angles of the light that can be received in the horizontal and vertical directions are the same. Under this premise, the size and area of the overlapped part of the collected infrared image and low-light image can be calculated. The target surface imaged by infrared low light is 2/3 the size of the target surface, and the pixels of the target surface are 640*480.

From the following formula, it can be concluded that the overlapping part of the infrared image and the low-light image accounts for the size of the entire image pixel:

In formula (1), u is the distance between the target and the objective lens, b is the distance between the two objective lenses in the parallel optical axis, and θ is the horizontal field of view of the horizontal lens.

Due to the method of parallel optical axis, there are differences in the spatial position of the image receiving source, and there are non-common parts in the infrared image and the low-light image. The pixel width and height of the light image are basically the same. The image clipping process is shown in Figure 4. First, extract the data, count the input data, including the number of rows and columns, determine the clipping range of the input image, and determine whether each input data is within the clipping range. When the data is within the cutting range, the data is input into the frame buffer, and when the data is not within the cutting range, the data is not processed, and the judgment of the next data is started.

When the rest of the two images are common parts, the two images are scaled so that the aspect ratios of the two images are the same, and the pixel coordinates of the two images can be kept the same to the maximum extent. The image scaling process is shown in Figure 5, which consists of a data buffer unit, a bilinear interpolation operation unit, and a coefficient generation and logic control unit. The data buffer unit uses two FIFOs for data line buffering, and uses ping-pong operation for data line buffer writing; when performing data interpolation operations, the logic control unit controls the data read-in and write-out of the data buffer unit, and generates data at the same time. The parameters of the interpolation are input into the bilinear interpolation operation unit. The bilinear interpolation operation unit performs interpolation in the Y direction for the two input line buffer data lines. After two interpolations in the Y direction, the Y direction can be obtained. The two results of the upper interpolation are performed, and then the interpolation in the X direction is performed, and the interpolation data in the X direction can be obtained by performing the interpolation in the X direction once. When the image is enlarged, the pixel data coordinates (Xnew, Ynew) of the new image can be calculated according to the magnification of the original image. The coordinates (X0, Y0) in the original image, (X0, Y0) are likely to be decimals, according to The four data closest to the coordinates (X0, Y0) in the original image are taken as the difference objects, and the value F of the new data can be inferred according to the distance between the coordinates.

To sum up, by adopting the image registration method mainly based on parallel optical axis image registration and supplemented by digital image registration, the digital image registration only needs to cut and zoom the image, which can greatly to improve the speed of image processing and improve the real-time performance of image processing.

Image fusion unit: used to fuse the two registered images by using the MIT pseudo-color image fusion algorithm, and output the fused color image; in the embodiment of the present application, the image fusion algorithm framework is shown in FIG. 6 convolvers, 4 dividers, 4 normalization processes and a delay output unit implement the MIT fusion algorithm. Since FPGAs are relatively simple to handle addition and subtraction, adders and subtractors are not included in the figure in this application. Since this application is a pixel-level image fusion, it is required that infrared pixels and low-light pixels correspond one-to-one, and the infrared image and low-light image need to be processed at the same time. The image is extracted from the frame buffer. The extracted images are respectively entered into the corresponding convolvers, convolved through the corresponding templates, and the convolution results are input into the divider, which is finally normalized by the normalization processing program. Processed so that the result is in the range 0 to 256 and is exactly 8-bit binary, and the image is enhanced in the process. After that, image fusion is performed. The input data of image fusion is two images, and the fusion process is basically the same as the above-mentioned image enhancement process. Finally, the output results are mapped to the three RGB channels respectively, and the fused color image can be obtained.

To sum up, this application uses FPGA to realize the pseudo-color image fusion algorithm of MIT. Due to the operation parallelism of FPGA, the image fusion speed will be greatly improved, the image output delay is very low, and the real-time performance is very strong. Second, MIT After the algorithm is fused, it is a color image, and the image has a stronger sense of hierarchy, which is more conducive to the observation and judgment of the image.

A/D conversion module: used to convert the image data output by the FPGA chip into a VGA analog signal using the A/D conversion chip, and display it through the display module; in the embodiment of the application, the display module is a liquid crystal display, which displays the resolution of the image The ratio is preferably 1920*1080.

External memory module: It is mainly divided into two parts. One is the memory chip SDRAM chip that can be used as a buffer area in the image processing process. As a frame buffer device, the real-time image is cached, and the real-time data interaction with the FPGA is performed; the second is A memory chip FLASH chip that stores some information that needs to be powered off and not lost, and stores FPGA configuration information.

External control module: adjust the internal mode of the system through the button, and control the output of different mode signals to cope with the use in different occasions.

Please refer to FIG. 7 , which is a flowchart of an infrared low-light image fusion method according to an embodiment of the present application. The infrared low-light image fusion method according to the embodiment of the present application includes the following steps:

Step 100: collect infrared images and low-light images respectively;

In step 100, the infrared image is collected by using the infrared sensor core of the UWA384CX-H42 model and equipped with an infrared lens. There is a filter film on the surface of the lens, which reflects visible light but allows infrared wavelengths of light to pass through, thereby reducing the passage of visible light and increasing the transmittance of infrared light. The IRFPA array (infrared focal plane array) that is projected on the back after converging by the lens, the IRFPA array senses the infrared light and performs imaging. Low-light images are collected by 1XC18/18WHS-CL model image intensifier movement and adjustable aperture optical lens. The output is PAL video mode. The external light information is collected in the photosensitive array after passing through the optical lens. Convergence and enhancement of the image on the focal plane increase the light sensitivity of the system and enhance low-light imaging.

Step 200: Decode the collected infrared images and low-light images from analog signals into digital signals through a multi-channel video decoding chip, and perform frame synchronization and frame buffer processing on the two-channel digital signals, and then output them to the FPGA processing module;

In step 200, since the image signals output by the image acquisition devices such as the infrared optical lens, the visible light optical lens, the infrared image detector and the low-light image detector are all output in the PAL (Phase Alteration Line, Pal system) format, the subsequent FPGA chip It cannot be processed directly, so it is necessary to use a video decoding chip to convert infrared images and low-light images from video analog signals to digital signals, and then send them to the FPGA for processing.

Step 300: filter, image registration and image fusion operations are performed on the infrared image and the low-light image by the FPGA processing module to obtain a fused color image;

In step 300, the image registration is specifically: performing optical registration, image clipping and image scaling operations on the filtered infrared image and the low-light image; the present application adopts a dual-channel parallel optical axis system to perform the front-end optical system registration. It will not interfere with each other, which is conducive to the further fusion of images. As shown in Figure 3, it is the optical path diagram of the parallel optical axis system. Since the horizontal and vertical field angles of the infrared lens and the low-light lens are not much different, it can be approximately considered that the elevation angles of the light that can be received in the horizontal and vertical directions are the same. The size and area occupied by the overlapping portion of the acquired infrared and low-light images. The target surface imaged by infrared low light is 2/3 the size of the target surface, and the pixels of the target surface are 640*480.

From the following formula, it can be concluded that the overlapping part of the infrared image and the low-light image accounts for the size of the entire image pixel:

In formula (1), u is the distance between the target and the objective lens, b is the distance between the two objective lenses in the parallel optical axis, and θ is the horizontal field of view of the horizontal lens.

Due to the method of parallel optical axis, there are differences in the spatial position of the image receiving source, and there are non-common parts in the infrared image and the low-light image. The pixel width and height of the light image are basically the same. The image clipping process is shown in Figure 4. First, extract the data, count the input data, including the number of rows and columns, determine the clipping range of the input image, and determine whether each input data is within the clipping range. When the data is within the cutting range, the data is input into the frame buffer, and when the data is not within the cutting range, the data is not processed, and the judgment of the next data is started.

When the rest of the two images are common parts, the two images are scaled so that the aspect ratios of the two images are the same, and the pixel coordinates of the two images can be kept the same to the maximum extent. The image scaling process is shown in Figure 5, which consists of a data buffer unit, a bilinear interpolation operation unit, and a coefficient generation and logic control unit. The data buffer unit uses two FIFOs for data line buffering, and uses ping-pong operation for data line buffer writing; when performing data interpolation operations, the logic control unit controls the data read-in and write-out of the data buffer unit, and generates data at the same time. The parameters of the interpolation are input into the bilinear interpolation operation unit. The bilinear interpolation operation unit performs interpolation in the Y direction for the two input line buffer data lines. After two interpolations in the Y direction, the Y direction can be obtained. The two results of the upper interpolation are performed, and then the interpolation in the X direction is performed, and the interpolation data in the X direction can be obtained by performing the interpolation in the X direction once. When the image is enlarged, the pixel data coordinates (Xnew, Ynew) of the new image can be calculated according to the magnification of the original image. The coordinates (X0, Y0) in the original image, (X0, Y0) are likely to be decimals, according to The four data closest to the coordinates (X0, Y0) in the original image are taken as the difference objects, and the value F of the new data can be inferred according to the distance between the coordinates.

In the embodiment of the present application, the image fusion is specifically: using the MIT pseudo-color image fusion algorithm to fuse the registered two images; the image fusion algorithm framework is shown in FIG. The MIT fusion algorithm is implemented with a processor, 4 normalization processes and a delay output unit. Since FPGAs are relatively simple to handle addition and subtraction, adders and subtractors are not included in the figure in this application. Since this application is a pixel-level image fusion, it is required that infrared pixels and low-light pixels correspond one-to-one, and the infrared image and low-light image need to be processed at the same time. The image is extracted from the frame buffer. The extracted images are respectively entered into the corresponding convolvers, convolved through the corresponding templates, and the convolution results are input into the divider, which is finally normalized by the normalization processing program. Processed so that the result is in the range 0 to 256 and is exactly 8-bit binary, and the image is enhanced in the process. After that, image fusion is performed. The input data of image fusion is two images, and the fusion process is basically the same as the above-mentioned image enhancement process. Finally, the output results are mapped to the three RGB channels respectively, and the fused color image can be obtained.

Step 400: Use an A/D conversion chip to convert the fused video data output by the FPGA chip into a VGA analog signal, and display it on a display.

In step 400, the display is a liquid crystal display, and the resolution of the displayed image is preferably 1920*1080.

FIG. 8 is a schematic structural diagram of a hardware device of the infrared low-light image fusion method provided by the embodiment of the present application. As shown in Figure 8, the device includes one or more processors and memory. Taking a processor as an example, the device may further include: an input system and an output system.

The processor, the memory, the input system and the output system may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 8 .

As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor executes various functional applications and data processing of the electronic device by running the non-transitory software programs, instructions and modules stored in the memory, that is, the processing method of the above method embodiment is implemented.

The memory may include a stored program area and a stored data area, wherein the stored program area can store an operating system and an application program required by at least one function; the stored data area can store data and the like. Additionally, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory may optionally include memory located remotely from the processor, which may be connected to the processing system via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input system can receive input numerical or character information and generate signal input. The output system may include a display device such as a display screen.

The one or more modules are stored in the memory, and when executed by the one or more processors, perform the following operations of any of the foregoing method embodiments:

Step a: collecting infrared images and low-light images respectively;

Step b: decode and convert the infrared image and the low-light image from the analog signal into a digital signal by using a multi-channel video decoding chip, and perform frame synchronization and frame buffer processing on the two-channel digital signal, and then output to the FPGA processing module;

Step c: After preprocessing, image registration and image fusion operations are performed on the infrared image and the low-light image through the FPGA chip, the fused color image is output;

Step d: use an A/D conversion chip to convert the color image into a VGA analog signal, and display it through a display module.

The above product can execute the method provided by the embodiments of the present application, and has functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, reference may be made to the method provided in this embodiment of the present application.

An embodiment of the present application provides a non-transitory (non-volatile) computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions can perform the following operations:

Step a: collecting infrared images and low-light images respectively;

Step b: decode and convert the infrared image and the low-light image from the analog signal into a digital signal by using a multi-channel video decoding chip, and perform frame synchronization and frame buffer processing on the two-channel digital signal, and then output to the FPGA processing module;

Step c: After preprocessing, image registration and image fusion operations are performed on the infrared image and the low-light image through the FPGA chip, the fused color image is output;

Step d: use an A/D conversion chip to convert the color image into a VGA analog signal, and display it through a display module.

An embodiment of the present application provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer , which causes the computer to do the following:

Step a: collecting infrared images and low-light images respectively;

Step b: decode and convert the infrared image and the low-light image from the analog signal into a digital signal by using a multi-channel video decoding chip, and perform frame synchronization and frame buffer processing on the two-channel digital signal, and then output to the FPGA processing module;

Step c: After preprocessing, image registration and image fusion operations are performed on the infrared image and the low-light image through the FPGA chip, the fused color image is output;

Step d: use an A/D conversion chip to convert the color image into a VGA analog signal, and display it through a display module.

The infrared low-light image fusion system, method and electronic device according to the embodiments of the present application design a collection and fusion system for infrared and low-light images through an infrared low-light image acquisition device and an FPGA data processing control device. Compared with the prior art, the present application At least have the following advantages:

1. In the image registration part, the image registration method is mainly used for parallel optical axis image registration, supplemented by digital image registration. When performing digital image registration, only the image needs to be cut and zoomed, which can be extremely The speed of image processing is greatly improved, and the real-time performance of image processing will be greatly improved;

2. The image fusion part uses FPGA to implement MIT's pseudo-color image fusion algorithm. Due to the characteristics of FPGA's operation parallelism, the image fusion speed will be greatly improved, the image output delay is very low, and the real-time performance is very strong. Secondly, the MIT algorithm fusion The latter is a color image, and the image has a stronger sense of hierarchy, which is more conducive to the observation and judgment of the image;

3. The entire image fusion system uses FPGA as the core chip for signal control and operation, and the circuit system is simple to build without the assistance of other processors, which is easy to implement.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined in this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (8)

1. An infrared low-light image fusion system, comprising:

the infrared image acquisition module: the infrared image acquisition module is used for acquiring an infrared image;

the low-light-level image acquisition module: used for collecting low-light level images;

the video data conversion module: the multi-channel video decoding chip is used for decoding the infrared image and the low-light-level image from analog signals to digital signals, and outputting the two paths of digital signals to the FPGA processing module after frame synchronization and frame buffer processing;

an FPGA processing module: the system is used for outputting a fused color image after preprocessing, image registration and image fusion operations are carried out on the infrared image and the low-light-level image through an FPGA chip;

an A/D conversion module: the A/D conversion chip is used for converting the color image into a VGA analog signal and displaying the VGA analog signal through the display module;

the FPGA processing module specifically comprises:

an image preprocessing unit: the infrared image and the low-light image are subjected to filtering processing;

an image registration unit: the image processing device is used for carrying out optical registration, image shearing and image scaling operation on the filtered infrared image and the low-light-level image;

the image registration unit performs optical registration, image shearing and image scaling operations on the infrared image and the low-light-level image, and specifically comprises the following steps:

carrying out image registration by adopting a dual-channel parallel optical axis system; assuming that the infrared lens and the low-light-level lens can receive light in the horizontal direction and the vertical direction at the same elevation angle, calculating the size and the area occupied by the overlapped part of the acquired infrared image and the acquired low-light-level image according to the elevation angle:

in the above formula, u is the distance between the target and the objective lens, b is the distance between the two objective lenses in the parallel optical axis, and θ is the horizontal field angle of the horizontal lens;

shearing the non-shared parts in the infrared image and the low-light-level image, firstly extracting data, counting input data, including the number of rows and the number of columns, determining the shearing range of the input image, judging whether each input data is in the shearing range, inputting the data into a frame buffer memory when the input data is in the shearing range, and starting to judge the next data when the input data is not in the shearing range;

when the rest of the two images are the common parts, carrying out zooming processing on the two images, wherein the zooming processing on the two images is realized by a data cache unit, a bilinear interpolation operation unit and a coefficient generation and logic control unit together; the data cache unit adopts two FIFOs to perform line cache of data and adopts ping-pong operation to perform line cache write-in of the data; when the interpolation operation of the data is carried out, the logic control unit controls the data reading and writing of the data cache unit, simultaneously generates interpolation parameters and inputs the interpolation parameters into the bilinear interpolation operation unit, the bilinear interpolation operation unit carries out interpolation in the Y direction on two input cache data lines, two interpolation results in the Y direction can be obtained after two times of interpolation in the Y direction, then the interpolation in the X direction is carried out, and the interpolation data in the X direction can be obtained after one time of interpolation in the X direction.

2. The infrared micro-optical image fusion system of claim 1, wherein the FPGA processing module further comprises:

an image fusion unit: and the image fusion module is used for performing fusion processing on the registered infrared image and low-light image by adopting an MIT pseudo-color image fusion algorithm and outputting a fused color image.

3. The infrared low-light-level image fusion system as claimed in claim 2, wherein the image fusion unit adopts an MIT pseudo-color image fusion algorithm to perform fusion processing on the two registered images, specifically: the MIT fusion algorithm is realized through 6 convolvers, 4 dividers, 4 normalization processes and a delay output unit; extracting images from an infrared image frame cache and a low-light-level image frame cache respectively in a synchronous extraction mode, enabling the extracted images to enter corresponding convolvers respectively for convolution, inputting convolution results into a divider, and finally performing normalization processing by a normalization process to enable the interval of the results to be between 0 and 256, so that the images are enhanced; and then inputting the infrared image and the low-light-level image for image fusion, and finally mapping the fusion result to the RGB three channels respectively to obtain a fused color image.

4. An infrared micro-optical image fusion system according to any one of claims 1 to 3, further comprising:

an external storage module: the image processing method comprises a storage chip SDRAM chip and a storage chip FLASH chip, wherein the storage chip SDRAM chip is used for caching real-time images and performing real-time data interaction with an FPGA chip in the image processing process;

an external control module: the FPGA chip is used for adjusting the internal mode of the FPGA chip through the button and controlling and outputting different mode signals.

5. An infrared low-light image fusion method is characterized by comprising the following steps:

step a: respectively collecting an infrared image and a low-light image;

step b: the infrared image and the low-light-level image are decoded and converted into digital signals from analog signals through a multi-channel video decoding chip, and the two paths of digital signals are subjected to frame synchronization and frame buffer processing and then output to an FPGA processing module;

step c: preprocessing, image registration and image fusion operations are carried out on the infrared image and the low-light-level image through an FPGA chip, and then a fused color image is output;

step d: converting the color image into a VGA analog signal by using an A/D conversion chip, and displaying the VGA analog signal by using a display module;

in the step c, the preprocessing, image registration and image fusion operations performed on the infrared image and the low-light-level image by the FPGA chip specifically include:

step c 1: filtering the infrared image and the low-light-level image;

step c 2: carrying out optical registration, image shearing and image scaling operation on the filtered infrared image and the low-light-level image;

in step c2, the performing optical registration, image cropping and image scaling operations on the infrared image and the low-light-level image specifically includes: carrying out image registration by adopting a dual-channel parallel optical axis system; assuming that the infrared lens and the low-light-level lens can receive light in the horizontal direction and the vertical direction at the same elevation angle, calculating the size and the area occupied by the overlapped part of the acquired infrared image and the acquired low-light-level image according to the elevation angle:

in the above formula, u is the distance between the target and the objective lens, b is the distance between the two objective lenses in the parallel optical axis, and θ is the horizontal field angle of the horizontal lens;

shearing the non-shared parts in the infrared image and the low-light-level image, firstly extracting data, counting input data, including the number of rows and the number of columns, determining the shearing range of the input image, judging whether each input data is in the shearing range, inputting the data into a frame buffer memory when the input data is in the shearing range, and starting to judge the next data when the input data is not in the shearing range;

when the rest of the two images are the common parts, carrying out zooming processing on the two images, wherein the zooming processing on the two images is realized by a data cache unit, a bilinear interpolation operation unit and a coefficient generation and logic control unit; the data cache unit adopts two FIFOs to perform line cache of data and adopts ping-pong operation to perform line cache write-in of the data; when the interpolation operation of the data is carried out, the logic control unit controls the data reading and writing of the data cache unit, simultaneously generates interpolation parameters and inputs the interpolation parameters into the bilinear interpolation operation unit, the bilinear interpolation operation unit carries out interpolation in the Y direction on two input cache data lines, two interpolation results in the Y direction can be obtained after two times of interpolation in the Y direction, then the interpolation in the X direction is carried out, and the interpolation data in the X direction can be obtained after one time of interpolation in the X direction.

6. The infrared low-light image fusion method according to claim 5, wherein in the step c, the pre-processing, image registration and image fusion operations of the infrared image and the low-light image by the FPGA chip further comprise:

step c 3: an image fusion unit: and the image fusion module is used for performing fusion processing on the registered infrared image and low-light image by adopting an MIT pseudo-color image fusion algorithm and outputting a fused color image.

7. The infrared low-light image fusion method according to claim 6, wherein in the step c3, the fusion process of the registered infrared image and low-light image by using the MIT false-color image fusion algorithm is specifically: the MIT fusion algorithm is realized through 6 convolvers, 4 dividers, 4 normalization processes and a delay output unit; extracting images from an infrared image frame cache and a low-light-level image frame cache respectively in a synchronous extraction mode, enabling the extracted images to enter corresponding convolvers respectively for convolution, inputting convolution results into a divider, and finally performing normalization processing by a normalization process to enable the interval of the results to be between 0 and 256, so that the images are enhanced; and then inputting the infrared image and the low-light-level image for image fusion, and finally mapping the fusion result to the RGB three channels respectively to obtain a fused color image.

8. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of infrared micro-light image fusion of any one of the preceding claims 5 to 7.

Images