JP7614670B2 - Fast image registration method and apparatus for multispectral cameras - Google Patents

Description

The present invention belongs to the field of multispectral image processing, and in particular to a method and apparatus for high-speed image registration for multispectral cameras.

Traditional single-sensor video cameras collect images in the Bayer format, with each pixel point collecting only one of the three components of RGB, and the other two components undergo interpolation estimation based on the information of nearby pixel points, so that there is a certain color distortion in the actually captured image. In order to obtain an image formation effect with high color reproducibility, multispectral cameras use a spectral prism system to divide light into multiple bands, which are collected by multi-sensors and image-formed separately. Due to the influence of the physical image formation light path, there exists a parameterizable offset between the subspectral band images, which requires registration to fuse and generate a true color image. Although the lens correction system can greatly improve the initial offset between the subspectral band images, in the process of using the camera, secondary offsets will occur in the subspectral band images after correction due to actual uncontrollable factors such as physical vibration and temperature changes. Such secondary offsets occur randomly, have a large occurrence probability, and are difficult to completely eliminate by the correction system. In order to ensure the color reproducibility of the generated image, it is necessary to perform rapid calibration with the assistance of software algorithms.

Registration of subspectral band images belongs to the scope of multispectral image registration. Unlike general image registration, there are local brightness and contrast differences between different spectral band images, making it difficult to directly register multispectral images using image brightness. Currently, in the academic world, measures such as mutual information and local normalized correlation coefficient are used as objective functions to optimize the registration parameters between two band images. The calculation of such measures is complex and time-consuming, making them difficult to be adopted in industry.

To solve the problem of high-speed image registration of the sub-spectral band of a multispectral camera, the present invention provides a method and apparatus for high-speed image registration for a multispectral camera. The present invention uses the normalized total gradient as a measure for solving the registration parameters, which is calculated based on the image gradient feature, and has the characteristics of low calculation cost and high speed. However, the normalized total gradient target has a problem of a small convergence region, and requires relatively good initial value parameters to quickly converge the optimization result to the optimal solution. Therefore, the present invention uses a grid block-based offset estimation and a global registration parameter initial value fitting technique based on a non-smooth grid offset to quickly solve the registration parameter initial value. The initial value parameters are then sent to a registration parameter optimizer based on the normalized total gradient to fine-tune and optimize the registration parameters. In order to reduce the influence of local brightness and contrast differences in the multispectral image and ensure the accuracy of the grid block offset estimation, the grid block offset estimation uses a phase correlation algorithm based on the robust gradient feature. Compared with similar registration methods, the method of the present invention has higher registration accuracy and faster registration speed.

The object of the present invention is achieved by the following technical solutions:

According to a first aspect of the present specification, a method for fast image registration for a multispectral camera is provided, the method comprising the steps of: selecting one band of a multispectral band image collected by a multispectral camera as a reference image, and the remaining bands as images to be registered, and calculating robust gradient features of the reference image and the image to be registered, respectively; blocking the robust gradient feature maps of the reference image and the image to be registered with a grid, and selecting a non-smooth grid; calculating an offset amount of the non-smooth grid by a phase correlation algorithm; fitting an initial global registration parameter value based on the grid offset amount of the non-smooth grid; sending the initial registration parameter value to an optimizer based on a normalized total gradient, and iterating through a gradient descent method to obtain accurate registration parameters; and applying the registration parameters obtained in step S5 to the image to be registered to obtain an accurately registered image result.

（ｘ，ｙ）は画像のグローバル座標であり、ｐは指数パラメータであり、ｆ_x（ｘ，ｙ），ｆ_y（ｘ，ｙ）は、それぞれ参照画像ｆ（ｘ，ｙ）のｘ、ｙ方向に沿う勾配であって、参照画像を畳み込みカーネルで畳み込むことによって計算されて得られ、 レジストレーション対象画像をｆ′（ｘ，ｙ）と記し、ｆ′（ｘ，ｙ）のロバスト勾配特徴マップｇ′（ｘ，ｙ）の計算表現式は以下のとおりであり、

ｆ_x′（ｘ，ｙ），ｆ_y′（ｘ，ｙ）は、それぞれレジストレーション対象画像ｆ′（ｘ，ｙ）のｘ、ｙ方向に沿う勾配であって、参照画像を畳み込みカーネルで畳み込むことによって計算されて得られる。 Furthermore, step S1 is specifically: the reference image is denoted as f(x, y), and the calculation expression of the robust gradient feature map g(x, y) of f(x, y) is as follows:

where (x, y) are the global coordinates of the image, p is an exponential parameter, f _x (x, y) and f _y (x, y) are the gradients along the x and y directions of the reference image f(x, y), respectively, which are calculated by convolving the reference image with a convolution kernel. The image to be registered is denoted as f′(x, y), and the calculation formula of the robust gradient feature map g′(x, y) of f′(x, y) is as follows:

fx _' (x,y) and _fy '(x,y) are the gradients along the x and y directions of the image f'(x,y) to be registered, respectively, and are calculated by convolving the reference image with a convolution kernel.

Furthermore, in step S2, the reference robust gradient feature map and the registration target robust gradient feature map obtained in step S1 are blocked using a regular grid.

Furthermore, in step S2, the selection of the non-smooth grid is specifically as follows: in sub-step S21, a training dataset of robust gradient feature grid images is created, i.e., a robust gradient feature grid image set is constructed, and categories are labeled for the grid images, where the smooth grid image is labeled as category 0 and the non-smooth grid image is labeled as category 1; in sub-step S22, a smooth and non-smooth grid classifier is trained; and in sub-step S23, the grid is classified by the trained classifier, and a non-smooth grid is selected.

Furthermore, in step S3, the offset amount is specifically calculated as follows: Sub-step S31 performs a two-dimensional Fourier transform on the reference robust gradient feature map and the registration target robust gradient feature map of the non-smooth grid obtained in step S2 to obtain the two-dimensional Fourier transform results of the reference grid image and the registration target grid image; Sub-step S32 calculates a conventionalized cross power spectral density using the two-dimensional Fourier transform result; Sub-step S33 performs a two-dimensional inverse Fourier transform on the conventionalized cross power spectral density to obtain a cross-correlation feature map; and Sub-step S34 obtains the local coordinates corresponding to the maximum cross-correlation value as the offset amount of the non-smooth grid.

と記し、ステップＳ３で得られた対応するオフセット量を

とし、求めるべきレジストレーションパラメータ初期値行列を

と記し、

は実数体を示し、座標変換関係に基づいて次のようになり、

すべての非平滑化グリッドに対して上記座標変換関係を応用し、行列方程

を取得し、Ｄはグリッド中心のグローバル座標行列であり、

はオフセット量行列であり、 サブステップＳ４２では、行列方程を解いて、レジストレーションパラメータ初期値行列

を取得し、Ｔは転置操作を示し、

は逆行列を求める操作を示す。 Further, in step S4, fitting the initial global registration parameters includes the following sub-steps: in sub-step S41, construct a linear equation system for the registration parameters and calculate the global coordinates of the i-th non-smooth grid center as

and the corresponding offset amount obtained in step S3 is written as

The registration parameter initial value matrix to be calculated is

He wrote,

denotes the field of real numbers, and based on the coordinate transformation relations,

Apply the above coordinate transformation relationship to all non-smooth grids and solve the matrix equation

where D is the grid-centered global coordinate matrix,

is an offset amount matrix, and in sub-step S42, the matrix equation is solved to obtain the registration parameter initial value matrix

where T denotes the transposition operation,

indicates the operation of finding the inverse matrix.

Furthermore, step S5 is specifically as follows: In sub-step S51, the registration parameter initial values obtained in step S4 are set as the initial parameter of an optimizer based on the normalized total gradient; in sub-step S52, an optimization problem is constructed using the normalized total gradient of the reference image and the image to be registered as the objective function, and iterative optimization of the registration parameters is performed using the gradient descent method; and in sub-step S53, the optimized registration parameters are output.

を計算し、上付き文字ｔはｔ回目の反復を示し、

はｔ回目の反復のパラメータ調整量を示し、 サブステップＳ５２２では、現在のレジストレーションパラメータ

をレジストレーション対象画像ｆ′（ｘ，ｙ）に応用し、（ｘ，ｙ）は画像のグローバル座標であり、レジストレーション対象画像ｆ′（ｘ，ｙ）に対して座標変換を行って

を取得し、

はレジストレーション対象画像の座標変換後の新座標であり、 サブステップＳ５２３では、座標変換後のレジストレーション対象画像

と参照画像ｆ（ｘ，ｙ）の正規化全勾配のレジストレーションパラメータに対する勾配を計算して、パラメータ調整量

を取得し、 サブステップＳ５２４では、サブステップＳ５２１に戻り、反復終了条件を満たすまで続け、前記反復終了条件は反復回数が設定された最大反復回数を超えることである。 Further, sub-step S52 is specifically as follows: In sub-step S521, the current registration parameters

where the superscript t indicates the t-th iteration,

indicates the parameter adjustment amount for the t-th iteration, and in substep S522, the current registration parameters

is applied to the registration target image f'(x, y), where (x, y) is the global coordinate of the image, and a coordinate transformation is performed on the registration target image f'(x, y) to obtain

Get

are new coordinates after the coordinate transformation of the registration target image, and in sub-step S523, the registration target image after the coordinate transformation is

Calculate the gradient of the normalized total gradient of the reference image f(x, y) with respect to the registration parameters, and

In sub-step S524, the process returns to sub-step S521 and continues until an iteration end condition is met, which is that the iteration number exceeds a set maximum iteration number.

Furthermore, step S6 is specifically as follows: in sub-step S61, coordinate transformation is performed on the registration target image based on the registration parameters obtained in step S5 to obtain the target pixel position, and in sub-step S62, the final output image is obtained using a bilinear interpolation algorithm based on the target pixel position.

According to a second aspect of the present specification, there is provided a high-speed image registration device for a multispectral camera, the device including a memory and one or more processors, executable code stored in the memory, the processor being used to implement the high-speed image registration method for a multispectral camera described in the first aspect when executing the executable code.

The present invention has the following beneficial effects:

1. The present invention utilizes a robust gradient feature map of gridding to effectively solve the problem of rapid estimation of initial registration parameters for multispectral camera images.

2. We use an improved normalized total gradient based registration parameter optimizer to solve the fast refinement registration problem of multispectral images.

FIG. 1 is a flow chart of a method for fast image registration provided by an exemplary embodiment. FIG. 2 is a block diagram of a method for fast image registration provided by an example embodiment. FIG. 3 is a block diagram of a regularized total gradient optimizer provided by an example embodiment. 4 and 5 are schematic diagrams of the non-registration and registration effects, respectively, provided by exemplary embodiments. 6 and 7 are schematic diagrams of the non-registration and registration effects, respectively, provided by other exemplary embodiments. 8, 9 and 10 are the original image, the robust gradient feature map and the effect of non-smooth regions, respectively, of an exemplary embodiment. FIG. 11 is a structural diagram of a high-speed image registration apparatus for multispectral cameras provided by an exemplary embodiment.

In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the drawings.

As is clear, the described embodiments are merely some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments of the present application, all other embodiments that a person skilled in the art can obtain without creative labor fall within the scope of protection of the present application.

A multispectral camera uses a spectroscopic device to split the input visible light into multiple spectral bands, and then collects full-resolution sub-spectral band images with independent CCD sensors. Since the physical image-forming optical path is affected by uncontrollable factors such as temperature change, vibration, etc. in the actual usage environment, there exists a parameterizable offset between the sub-spectral band images. To estimate the registration parameters, the present invention proposes a fast image registration method for multispectral cameras. As shown in Figures 1 and 2, the method includes the following steps S1 to S6.

In step S1, a multispectral camera collects target scene images to obtain multispectral band images. One of the band images is selected as a reference image, and the remaining band images are images to be registered. The robust gradient features of the reference image and the image to be registered are calculated, respectively.

（ｘ，ｙ）は画像のグローバル座標であり、ｐは指数パラメータであり、ｐは（０，１）を取り、本実施例でｐは０．５を取り、ｆ_x（ｘ，ｙ），ｆ_y（ｘ，ｙ）は参照画像ｆ（ｘ，ｙ）のｘ、ｙ方向に沿う勾配をそれぞれ示し、畳み込みカーネルで参照画像を畳み込むことによって計算することができ、具体的に、ｆ_x（ｘ，ｙ）はｘ方向のソーベル演算子

で参照画像ｆ（ｘ，ｙ）に対して２次元畳み込みを行って取得することができ、ｆ_y（ｘ，ｙ）はｙ方向のソーベル演算子

で参照画像ｆ（ｘ，ｙ）に対して２次元畳み込みを行って取得することができる。 In one embodiment, the method of calculating the robust gradient feature will be described in detail. The reference image is denoted as f(x,y), and the calculation expression of the robust gradient feature map g(x,y) of the reference image is as follows:

where (x, y) is the global coordinate of the image, p is an exponential parameter, p takes the value of (0, 1), and in this embodiment, p takes the value of 0.5. f _x (x, y) and f _y (x, y) denote the gradients of the reference image f(x, y) along the x and y directions, respectively, and can be calculated by convolving the reference image with a convolution kernel. Specifically, f _x (x, y) is the Sobel operator in the x direction.

The reference image f(x, y) can be obtained by performing two-dimensional convolution with f y (x, y), and f _y (x, y) is the Sobel operator in the y direction.

It can be obtained by performing two-dimensional convolution on the reference image f(x, y).

ｆ_x′（ｘ，ｙ），ｆ_y′（ｘ，ｙ）はレジストレーション対象画像ｆ′（ｘ，ｙ）のｘ、ｙ方向に沿う勾配をそれぞれ示し、同様に、畳み込みカーネルで参照画像を畳み込むことによって計算することができる。 The image to be registered is denoted as f′(x,y), and the calculation method of the robust gradient feature map g′(x,y) of the image to be registered is the same as that of the reference image, and the calculation formula is as follows:

fx _' (x,y) and _fy '(x,y) denote the gradients along the x and y directions, respectively, of the image to be registered f'(x,y), and can similarly be computed by convolving the reference image with a convolution kernel.

In step S2, the robust gradient feature maps of the reference image and the image to be registered are blocked into grids, and a non-smooth grid is selected.

の矩形であり、Ｈ，Ｗは画像の高さ及び幅をそれぞれ示し、すなわち画像を１００個のグリッド領域に規則的に分ける。 In one embodiment, the reference robust gradient feature map and the robust gradient feature map to be registered obtained in step S1 are blocked by a regular grid, and the grid has a size of

where H, W are the height and width of the image, respectively, i.e., the image is regularly divided into 100 grid regions.

In one embodiment, the non-smooth grid is selected in the following manner:

In substep S21, a training dataset of robust gradient feature grid images is created, i.e., a robust gradient feature grid image set is constructed, and categories are labeled for the grid images, with smooth grid images labeled as category 0 and non-smooth grid images labeled as category 1; in substep S22, smooth and non-smooth grid classifiers are trained with a generic classifier model; and in substep S23, the grids are classified with the trained classifiers and a non-smooth grid is selected.

In step S3, the offset of the non-smooth grid is calculated using a phase correlation algorithm.

In one embodiment, the calculation of the offset amount includes the following substeps S31 to S34:

を行って、参照グリッド画像及びレジストレーション対象グリッド画像の２次元フーリエ変換結果

及び

を取得し、 サブステップＳ３２では、コンベンショナル化クロスパワースペクトル密度

を計算し、

は

の共役結果を示し、 サブステップＳ３３では、

に対して２次元フーリエ逆変換

を行って、相互相関特徴マップ

を取得し、 サブステップＳ３４では、最大相互相関値に対応する局所座標

をｉ番目の非平滑化グリッドオフセット量として取得し、

はｉ番目の非平滑化グリッドの局所座標を示す。 In sub-step S31, the reference robust gradient feature map and the robust gradient feature map to be registered of the non-smooth grid obtained in S2 are denoted as p _i and q _i , respectively, where i denotes the i-th non-smooth grid.

The two-dimensional Fourier transform results of the reference grid image and the registration target grid image are

and

In sub-step S32, the conventionalized cross power spectral density

Calculate

teeth

In sub-step S33,

2D inverse Fourier transform for

Then, we obtain the cross-correlation feature map

In sub-step S34, the local coordinates corresponding to the maximum cross-correlation value are obtained.

is taken as the i-th non-smooth grid offset,

denotes the local coordinate of the i-th non-smooth grid.

In step S4, the initial global registration parameters are fitted based on the grid offset amount.

In one embodiment, the initial global registration parameters are fitted by the following steps:

と記し、ステップＳ３で得られた対応するオフセット量を

とし、求めるべきレジストレーションパラメータ初期値行列を

と記し、

は実数体を示し、座標変換関係に基づいて次のようになり、

すべての非平滑化グリッドに対して上記座標変換関係を応用し、行列方程

を取得することができ、 グリッド中心のグローバル座標行列は

であり、オフセット量行列は

であり、Ｋは非平滑化グリッドの個数である。 In substep S41, a system of linear equations is constructed for the registration parameters, and the global coordinates of the i-th non-smooth grid center are calculated as

and the corresponding offset amount obtained in step S3 is written as

The registration parameter initial value matrix to be calculated is

He wrote,

denotes the field of real numbers, and based on the coordinate transformation relations,

Apply the above coordinate transformation relationship to all non-smooth grids and solve the matrix equation

and the global coordinate matrix at the grid center is

and the offset matrix is

where K is the number of non-smooth grids.

を取得することができ、Ｔは転置操作を示し、

は逆行列を求める操作を示す。 In sub-step S42, the matrix equation is solved to obtain the registration parameter initial value matrix

where T denotes the transposition operation,

indicates the operation of finding the inverse matrix.

In step S5, the initial registration parameters are sent to an improved normalized total gradient-based optimizer and optimized to obtain accurate registration parameters.

In one embodiment, the optimal registration parameters are determined in the following manner, as shown in Figure 3:

を探して、

を最も小さくすることであり、ＮＴＧは正規化全勾配関数であり、

はレジストレーション対象画像の座標変換後の新座標であり、即ち、

、

であり、

は行列Ｍのｋ行目ｌ列目の元素値を示す。このとき、目的関数はＭに関する非凸実数値関数であり、

と簡単に示すことができ、勾配降下法で反復して最適なレジストレーションパラメータを求め、反復式は

であり、上付き文字ｔはｔ回目の反復を示し、

はｔ回目の反復のパラメータ調整量を示す。本発明は勾配降下法で反復して最適なレジストレーションパラメータを求め、その計算の複雑さがより低く、計算速度がより速い。具体的に、パラメータ調整量は

であり、

は毎回の反復ステップサイズを示し、本実施例で

を取り、

はｔ回目の反復で、正規化全勾配関数ＮＴＧのＭに関する導関数を示し、その計算式は以下のとおりであり、

は画素点（ｘ，ｙ）の重みであり、Ｌは座標行列であり、Ｔは転置操作を示し、

は座標変換後のレジストレーション対象画像

が画素点（ｘ，ｙ）で勾配を求めることを示し、

は参照画像ｆ（ｘ，ｙ）と座標変換後のレジストレーション対象画像

の差分画像を示し、

は差分画像

の画素点（ｘ，ｙ）での二次勾配特徴値であり、

は座標変換後のレジストレーション対象画像

の画素点（ｘ，ｙ）での二次勾配特徴値であり、

はロバスト関数であり、

、パラメータは

であり、

は差分画像

のｘ、ｙ方向に沿う勾配をそれぞれ示し、ｆ_x′，ｆ_y′は座標変換後のレジストレーション対象画像

のｘ、ｙ方向に沿う勾配をそれぞれ示し、

はｘ、ｙ方向の勾配操作の随伴演算子をそれぞれ示し、それぞれ演算子

及び

で画像に対して２次元畳み込みを行って取得することができる。 The normalized total gradient of the reference image and the image to be registered is the objective function to determine whether the images are registered or not. If the images are perfectly registered, the normalized total gradient between the reference image and the image to be registered is the smallest. The image to be registered is denoted as f'(x,y), and the objective of registration is to find a single optimal registration parameter

Looking for,

where NTG is the normalized total gradient function,

are the new coordinates after the coordinate transformation of the image to be registered, i.e.

,

and

indicates the element value in the kth row and lth column of the matrix M. In this case, the objective function is a non-convex real-valued function of M,

It can be easily shown that the optimal registration parameters are found by iteratively using gradient descent, and the iterative formula is

where the superscript t denotes the t-th iteration,

denotes the parameter adjustment amount in the t-th iteration. The present invention iteratively obtains the optimal registration parameters by gradient descent, which has lower calculation complexity and faster calculation speed. Specifically, the parameter adjustment amount is

and

indicates the iteration step size, and in this embodiment

Take

denotes the derivative of the normalized total gradient function NTG with respect to M at the t-th iteration, and its calculation formula is as follows:

is the weight of the pixel point (x, y), L is the coordinate matrix, and T denotes the transposition operation.

is the image to be registered after coordinate transformation

indicates that the gradient is found at pixel point (x, y),

is the reference image f(x, y) and the registration target image after coordinate transformation.

shows the difference image of

is the difference image

is the quadratic gradient feature value at pixel point (x, y) of

is the image to be registered after coordinate transformation

is the quadratic gradient feature value at pixel point (x, y) of

is a robust function,

, the parameters are

and

is the difference image

are the gradients in the _x and _y directions of the image to be registered after the coordinate transformation.

are the gradients along the x and y directions, respectively,

denote the adjoint operators of the gradient operations in the x and y directions, respectively, and the operators

and

It can be obtained by performing two-dimensional convolution on the image.

As described above, step S5 specifically includes the following substeps S51 to S53.

とし、初期化パラメータ調整量は

であり、 サブステップＳ５２では、参照画像とレジストレーション対象画像の正規化全勾配を目的関数として、最適化問題を構築し、勾配降下法でレジストレーションパラメータに対して反復最適化を行い、その具体的なステップは以下のサブステップＳ５２１～サブステップＳ５２４を含み、 サブステップＳ５２１では、現在のパラメータ

を計算し、 サブステップＳ５２２では、現在のパラメータ

をレジストレーション対象画像に応用し、レジストレーション対象画像ｆ′（ｘ，ｙ）に対して座標変換を行って、

を取得し、 サブステップＳ５２３では、座標変換後のレジストレーション対象画像

と参照画像ｆ（ｘ，ｙ）の正規化全勾配のレジストレーションパラメータに対する勾配

を計算して、パラメータ調整量

を取得し、 サブステップＳ５２４では、サブステップＳ５２１に戻り、反復終了条件を満たすまで続け、反復終了条件は反復回数が設定された最大反復回数Ｔを超えることであり、本実施例でＴ＝６であり、 サブステップＳ５３では、最適化後のレジストレーションパラメータを出力する。 In sub-step S51, the initial registration parameter values obtained in step S4 are normalized to the initial parameter values of the total gradient optimizer.

The amount of initialization parameter adjustment is

In sub-step S52, an optimization problem is constructed using the normalized total gradient of the reference image and the registration target image as an objective function, and the registration parameters are iteratively optimized by the gradient descent method. The specific steps include the following sub-steps S521 to S524: In sub-step S521, the current parameters

In substep S522, the current parameters

is applied to the registration target image, and a coordinate transformation is performed on the registration target image f′(x, y) to obtain

In sub-step S523, the registration target image after the coordinate transformation is obtained.

and the normalized total gradient of the reference image f(x, y) with respect to the registration parameters

Calculate the parameter adjustment amount

In sub-step S524, the process returns to sub-step S521 and continues until an iteration end condition is satisfied, which is that the iteration number exceeds a set maximum iteration number T, which is 6 in this embodiment. In sub-step S53, the optimized registration parameters are output.

In step S6, the optimized registration parameters are applied to the registration target image to obtain an accurately registered image result.

を取得し、 サブステップＳ６２では、目的画素位置

に基づいて、バイリニア補間アルゴリズムで最終的な出力画像を取得する。 In one embodiment, step S6 is specifically as follows: In sub-step S61, a coordinate transformation is performed on the registration target image f'(x, y) based on the registration parameters obtained in step S5 to obtain a target pixel position f'(x, y).

In sub-step S62, the target pixel position is obtained.

Based on this, a bilinear interpolation algorithm is used to obtain the final output image.

Explanation of the implementation effects

As shown in Table 1, after testing, the high-speed image registration method for multispectral cameras according to the present invention has advantages such as faster speed and higher accuracy compared to similar methods. The size of the image used during testing is 480 x 640 pixels, the processor used to run the algorithm is 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz 2.42GHz, and the memory is 16.0GB.

(Table 1) Comparison of test results

Figures 4 to 7 are schematic diagrams of the effects of implementing the present invention. Figures 4 and 5 are schematic diagrams of the effects of no registration and registration in one embodiment, respectively, and Figures 6 and 7 are schematic diagrams of the effects of no registration and registration in another embodiment, respectively. When calibrated using the high-speed registration method of the present invention, the problems of color shift and ghost images in color images synthesized by a multispectral camera are greatly improved.

Figures 8 to 10 are schematic diagrams of the output effects of the smoothed and non-smoothed grid classifiers of the present invention, where Figures 8, 9, and 10 respectively show the original image, robust gradient feature map, and non-smoothed region effects of one embodiment, and the classifiers selected the non-smoothed grid regions of the robust gradient feature map well.

In correspondence with the embodiment of the high-speed image registration method for a multispectral camera described above, the present invention further provides an embodiment of a high-speed image registration device for a multispectral camera.

Referring to FIG. 11, a high-speed image registration device for a multispectral camera provided by an embodiment of the present invention includes a memory and one or more processors, and executable code is stored in the memory. When the processor executes the executable code, it is used to implement the high-speed image registration method for a multispectral camera of the above embodiment.

The embodiment of the high-speed image registration device for multispectral cameras of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities may be a device or equipment such as a computer. The embodiment of the device may be realized by software, hardware, or a combination of software and hardware. For example, when it is realized by software, the processor of the device with data processing capabilities located therein reads the corresponding computer program instructions in the non-volatile memory into the memory and executes them as one device in the logical sense. In terms of hardware, as shown in FIG. 11, it is a hardware structure diagram of the device with data processing capabilities located in the device of the present invention. In addition to the processor, memory, network interface, and non-volatile memory shown in FIG. 11, the device with data processing capabilities located in the device of the embodiment may further include other hardware based on the actual function of the device with data processing capabilities, which will not be described again.

The process for implementing the functions and actions of each unit of the above device can be specifically referred to in the process for implementing the corresponding steps of the above method, and will not be described repeatedly here.

The device embodiments basically correspond to the method embodiments, so for the relevant parts, please refer to the description of the method embodiments. The device embodiments described above are merely exemplary, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one location or distributed among multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Those skilled in the art can understand and implement it without creative labor.

An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon, the program, when executed by a processor, implementing the high-speed image registration method for a multispectral camera of the above embodiment.

The computer readable storage medium may be an internal storage unit of any device with data processing capability described in any of the above embodiments, such as a hard disk or memory. The computer readable storage medium may be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart memory card, an SD card, a flash card, etc., disposed in the device. Furthermore, the computer readable storage medium may include an internal storage unit of any device with data processing capability and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the device with data processing capability, and can also be used to temporarily store output or output data.

Claims (8)

A high-speed image registration method for a multispectral camera, comprising the following steps S1 to S6:
In step S1, one band of a multi-spectral band image collected by a multi-spectral camera is selected as a reference image, and the remaining bands are taken as images to be registered, and robust gradient feature maps of the reference image and the image to be registered are calculated respectively;
In step S2, the robust gradient feature maps of the reference image and the registration target image are blocked with a grid, and a non-smooth grid is selected;
In step S3, the offset amount of the non-smooth grid is calculated by a phase correlation method ;
In the step S4, fitting an initial value of a global registration parameter based on a grid offset amount of a non-smooth grid, and fitting the initial value of the global registration parameter includes the following sub-steps S41 to S42:
In the sub-step S41, a system of linear equations for the registration parameters is constructed, and the global coordinates of the i-th non-smooth grid center are calculated as

and the corresponding offset amount obtained in step S3 is written as

The registration parameter initial value matrix to be calculated is

He wrote,

indicates the real number field, and based on the coordinate transformation relationship, it becomes as shown in the following Equation 3 :

(3)
Apply the above coordinate transformation relationship to all non-smooth grids and solve the matrix equation

where D is the grid-centered global coordinate matrix,

is the offset matrix,
In the sub-step S42, the matrix equation is solved to obtain the registration parameter initial value matrix

where T denotes the transposition operation,

indicates the operation of finding the inverse matrix,
In step S5, the initial registration parameters are sent to an optimizer based on a normalized total gradient method , and accurate registration parameters are obtained by iteratively performing a gradient descent method;
In step S6, the registration parameters obtained in step S5 are applied to the registration target image to obtain an accurately registered image result ;
Specifically, step S1 includes the following steps:
The reference image is denoted as f(x, y), and the calculation formula of the robust gradient feature map g(x, y) of f(x, y) is as follows:

(1)
where (x, y) are the global coordinates of the image, p is an exponential parameter, f _x (x, y) and f _y (x, y) are the gradients along the x and y directions of the reference image f(x, y), respectively, calculated by convolving the reference image with a convolution kernel;
The registration target image is denoted as f′(x, y), and the calculation expression of the robust gradient feature map g′(x, y) of the registration target image f′(x, y) is as follows:

(2)
f _x '(x,y) and f _y '(x,y) are the gradients along the x and y directions of the image f '(x,y) to be registered, respectively, and are calculated by convolving the reference image with a convolution kernel.
A method for fast image registration for multispectral cameras, comprising: 2. The method for high-speed image registration for multispectral cameras according to claim 1, wherein in step S2, the reference robust gradient feature map and the registration target robust gradient feature map obtained in step S1 are blocked using a regular grid. In step S2, the selection of the non-smooth grid specifically includes the following sub-steps S21 to S23:
In the sub-step S21, a training dataset of robust gradient feature grid images is created, i.e., a robust gradient feature grid image set is constructed, and categories are labeled for the grid images, where smooth grid images are labeled as category 0 and non-smooth grid images are labeled as category 1;
In the sub-step S22, smooth and non-smooth grid classifiers are trained;
The method for fast image registration for multispectral cameras according to claim 1, characterized in that in the sub-step S23, the grid is classified by a trained classifier and a non-smooth grid is selected. In step S3, the calculation of the offset amount specifically includes the following sub-steps S31 to S34:
In the sub-step S31, a two-dimensional Fourier transform is performed on the reference robust gradient feature map and the robust gradient feature map to be registered of the non-smooth grid obtained in the step S2 to obtain two-dimensional Fourier transform results of the reference robust gradient feature map and the robust gradient feature map to be registered;
In the sub-step S32, a conventionalized cross power spectral density is calculated using the two-dimensional Fourier transform result;
In the sub-step S33, a two-dimensional inverse Fourier transform is performed on the conventionalized cross power spectral density to obtain a cross-correlation feature map;
The method for high-speed image registration for multispectral cameras according to claim 1 , wherein in the sub-step S34, a local coordinate corresponding to a maximum cross-correlation value is obtained as an offset amount of the non-smooth grid. Specifically, step S5 includes the following sub-steps S51 to S53:
In the sub-step S51, the registration parameter initial values obtained in the step S4 are set as initial value parameters of an optimizer based on a normalized total gradient;
In the sub-step S52, an optimization problem is constructed using the normalized total gradient of the reference image and the registration target image as an objective function, and the registration parameters are iteratively optimized by the gradient descent method;
The method for high-speed image registration for multispectral cameras according to claim 1 , wherein in the sub-step S53, optimized registration parameters are output. Specifically, the sub-step S52 includes the following sub-steps S521 to S524:
In the sub-step S521, the current registration parameters

where the superscript t indicates the t-th iteration,

indicates the parameter adjustment amount for the t-th iteration,
In the sub-step S522, the current registration parameters

is applied to the registration target image f'(x, y), where (x, y) is the global coordinate of the image, and a coordinate transformation is performed on the registration target image f'(x, y) to obtain

Get

are the new coordinates after the coordinate transformation of the image to be registered,
In the sub-step S523, the registration target image after the coordinate transformation

Calculate the gradient of the normalized total gradient of the reference image f(x, y) with respect to the registration parameters, and

Get
The method for fast image registration for multispectral cameras according to claim 5 , wherein in the sub-step S524, the process returns to the sub-step S521 and continues until an iteration end condition is satisfied. Specifically, step S6 includes the following sub-steps S61 to S62:
In the sub-step S61, a coordinate transformation is performed on the registration target image based on the registration parameters to obtain a target pixel position;
The method for fast image registration for multispectral cameras according to claim 1 , wherein in the sub-step S62, a final output image is obtained by a bilinear interpolation algorithm based on the target pixel position. 1. A high speed image registration apparatus for a multispectral camera, comprising: a memory; and one or more processors, the memory having executable code stored therein, the apparatus comprising:
The apparatus for high-speed image registration for a multispectral camera, wherein the processor is adapted to implement the method for high-speed image registration for a multispectral camera according to any one of claims 1 to 7 when executing the executable code.

Images