JPS6119076B2 - - Google Patents

Description

Detailed Description of the Invention The present invention automatically searches for, captures, and tracks a desired target or scene within the field of view of the imaging device by correlation processing of image signals supplied from a television camera or an infrared imaging device. This relates to a device for

Conventionally, with this type of device, in order to search and capture a desired target or scene, an operator displays images supplied from an imaging device on a CRT, and identifies and determines the target by watching the CRT screen. This system had drawbacks such as large individual differences depending on the ability of the operator and the need to have replacement personnel on hand due to physical fatigue of the operator.

In addition, there is a time delay of about 1 second between when the operator issues a tracking mode command after searching and capturing a desired target or scene within the field of view of the imaging device. Switching to automatic tracking mode did not go smoothly.

In order to eliminate these drawbacks, the present invention divides the image supplied from the imaging device into α and β parts along the directions parallel and perpendicular to the scanning line, and in each divided area, a desired target is set in advance. Alternatively, two-dimensionally scan a reference screen that stores image signals of a scene, find the maximum value of the correlation function, and calculate their α×β
In order to shorten the processing time, i are selected from among the maximum values of the correlation coefficients in descending order of the processing time, and each i region is multiplied by A _H and A _V in the directions parallel and perpendicular to the scanning line.
The above reference screen is scanned two-dimensionally to find the maximum value of the correlation coefficient in the above i regions, and one of the above divided regions that gives the maximum correlation coefficient is selected and newly For j input correlated screens,
The reference screen is scanned two-dimensionally within the above divided area to find the maximum value of the correlation coefficients, and the number n _c of which the maximum value of these j correlation coefficients exceeds a preset threshold is counted. , whose count value n _c is n _t (n _t
<j) A larger target or scene is selected as the desired target or scene, and the tracking mode is automatically entered. The drawings will be explained in detail below.

FIGS. 1 to 3 are explanatory diagrams of target or scene searching and capturing operations in the correlation tracking device according to the present invention.

Figure 1 explains the arrangement of regions when the image supplied from the imaging device is divided into α and β regions along the directions parallel and perpendicular to the scanning line, and the scanning method of the reference screen within the regions. It is a diagram. Figure 2 shows the correlation coefficients obtained by the above-mentioned scanning method and i regions selected from the largest amplitude values, and A _H
FIG. 12 is a diagram showing correlation coefficients calculated by two-dimensionally scanning a reference screen for each region by multiplying by A _V and A V times. Figure 3 shows that from among the correlation coefficients calculated for the above i regions, one region with the maximum correlation coefficient is selected, and the calculation is performed for j newly input correlated images in this region. It is a figure showing a correlation coefficient.

Note that in the following, for convenience of explanation, i=3, j=
Case 5 will be explained.

In FIG. 1, 1 is a correlated screen supplied from an imaging device, 2 is a divided area, and 3 is a reference screen. In the following description, it is assumed that a required reference screen is selected from image signals of tracking targets or scenes stored in an image database in advance.

Note that the size of the above correlated screen 1 is K pixels x
An explanation will be given assuming that L pixels and the size of the reference screen 3 are M pixels×N pixels.

In a region 2 in which the correlated screen 1 is divided into α and β parts along the direction parallel and perpendicular to the scanning line, the reference screen 3 is divided into α and β pixels in the direction parallel to the scanning line and at an interval of K/αB _H pixels and along the scanning line. Two-dimensional coarse scanning is performed in the vertical direction at L/βB _V pixel intervals, and the correlation coefficient C(k,
Calculate l).

In this case, the image signal of the correlated screen 1 is Q(m+
k, n+l) and the image signal of reference screen 3 as P(m,
n), the correlation coefficient C(k,l) is given by equation (1).

Here, (m, n) is an orthogonal coordinate system in which the upper left end of the reference screen 3 is set as the coordinate origin, and (k, l) is the orthogonal coordinate system in which the upper left end of each divided area 2 is set as the coordinate origin, and k, l is a value that satisfies the conditions of equations (2) and (3).

1≦k≦(K/α−M+1) (2) 1≦l≦(K/β−N+1) (3) In this case, the calculation time T for α×β correlation coefficients as shown in Figure 2 is one time P(m,n) and Q
If the product calculation time of (m+k, n+l) is t, then the
Equation (4) is obtained.

As an example, K=L=256 pixels, M=N=
16 pixels, α=β=8, B _H =B _V =4, t=
When using a 70nsec multiplier, T = (8) ²・(16) ²・(3) ²・(70×10 ^-9 ) 〓10.3msec, which is 1 of the standard TV scanning system of 30 frames/sec. Can be processed within a frame.

Next, among the amplitudes of α×β correlation coefficients as shown in FIG.
In addition to selecting C ₁ , C ₂ , and C ₃ , in order to prevent the target from moving outside the above-mentioned divided area and not being able to correlate in the next input correlated screen, Expand the correlation calculation area by multiplying it by A _H and A _V in the vertical direction, scan the reference screen two-dimensionally in each area, and calculate the correlation coefficient C ₁
(k・l), C ₂ (k・l), and C ₃ (k・l).

In this case, the calculation time T' of the correlation coefficient is expressed by equation (5), where t is the time required for one product calculation.

T'=i・M・N・(A _H・K/α−M+1)・( _AV・L/βN+1)・t (5) As an example, K=L=256 pixels, M=N=16
Pixel, α = β = 8, A _H = A _V = 1.5, t = 70 nsec
^When using ^{a multiplier} of can.

Correlation coefficients C ₁ (k, l), C ₂ calculated as above
(k, l), C ₃ (k, l), select one region where the amplitude is maximum, and in this region (2),
As shown in the figure, the reference screen (3) is two-dimensionally scanned for five correlated images that are input continuously, and the correlation coefficients C ₍₁₎ (k, l), C _{(2 )} (k, l), C
₍₃₎ (k, l), C ₍₄₎ (k, l), C ₍₅₎ (k,
Calculate l).

Here, the maximum value of the above correlation coefficient is E ₁ , E ₂ ,
E ₃ , E ₄ , E ₅ , and a preset threshold value is E _t . These E ₁ to _{E 5} are compared with E _t , the number n _c exceeding E _t is counted, and this counted value n _c is the threshold value n _t
If the target exceeds the target, it is identified as the desired target and the system shifts to tracking mode. Such processing is performed when the reference image is selected from the image database and continuously used as described above, and the target image is translated, enlarged,
This is to prevent malfunctions due to changes in the maximum value of the correlation coefficient due to reduction, rotation, etc.

Note that if n _c < _nt = 3, the search operation shown in FIG. 1 is returned to and the above operation is repeated.

By the way, the continuously input j shown in Figure 3
= Calculation time of correlation coefficient for 5 correlated images
T″ is expressed by Equation (6), where t is the time required for one product operation.

T″=j・M・N・(A _H・K/α−M+1)・( _AV・L/β−N+1)・t (6) As an example, K=L=256 pixels, M=N=16
Pixel, α = β = 8, A _H = A _V = 1.5, t = 70 nsec
When using a multiplier, T″=5・(16) ²・(33) ²・(70×10 ^-9 ) 〓97.6msec, which can be processed within 3 frames of the standard television scanning system of 30 frames/sec.

In the above-described correlation tracking device according to the present invention, the time required to search and capture a target or scene is K.
=L=256 pixels, M=N=16 pixels, α=β=8,
A _H = A _V = 1.5, B _H = B _V = 4, i=3, j=
5. When t=70nsec, at most 6 frames (〓
0.2sec), and can automatically shift to tracking mode within the integration time of the naked eye (0.3sec).

Next, in the tracking operation, first, the correlation coefficient is similarly calculated within the area shown in FIG. 3, and the coordinates (k ₀ , l ₀ ) where the amplitude is maximized are determined, and these are determined as the tracking point. For the next input correlated screen 1, as shown in FIG.
The correlation coefficient C(k,l) is calculated by limiting the scanning area of the reference screen 3 to a rectangular area 4 of ε pixels.

Similarly, the coordinates (k ₁ , l ₁ ) that maximize the amplitude of this correlation coefficient C(k, l) are found, and the distance R from (ko, l ₀ ) is found from equation (7).

R=√｜ ₁ − ₀ | ² +｜ ₁ − ₀ | ² (7) Here, the above R is compared with the preset ε, and if R≧ε, as shown in Figure 4b, , a rectangular area 4 having a size of 2aR pixels×2aR pixels (a≧1) is provided centered on the tracking point (k ₁ , l ₁ ), and the scanning area of the reference screen 3 is limited to this area.

The calculation time Tc of the above correlation coefficient is expressed by equation (8), where t is the time required for one product calculation.

〓Tc=M・N・(2aR) ²・t: When R≧ε =M・N・(2ε) ²・t When R<ε (8) By the way, the maximum value of the above distance R is usually Under the operating conditions, the width of the correlated screen per second is less than the full width of the correlated screen, so as an example, the size of the correlated screen is K = L.
= 256 pixels, in the standard television scanning system of 30 frames/sec, the above maximum R is 8.5 pixels between frames (33.3 msec).

At this time, when tracking is performed by setting a reference screen of size M=N=16 pixels as a=2, and using a multiplier with t=70 nsec, the calculation time Tc is Tc=(16) ²・(2×2× 8.5) ²・(70×10 ^-9 ) 〓 20.7 msec, which is smaller than 33.3 msec, so it is possible to perform correlation tracking of standard television scanning images in real time.

In the correlation tracking device according to the present invention described above,
It automatically detects multiple targets or scenes in a complex background, selects the one with the highest correlation among them, and captures it by performing double threshold processing over j frames. As a result, a stable and highly reliable system can be constructed.

FIG. 5 is a block diagram of an embodiment of the present invention.

In FIG. 5, an image signal supplied from an imaging device 6 mounted on a servo mount 5 is converted into a digital signal through an analog-digital conversion circuit 7.

Using this image signal, first, a search for the target is performed as follows. That is, the area dividing circuit 8
The correlated screen is divided into α and β pieces along the parallel and perpendicular directions to the scanning line using
The image signals within these areas are stored in the buffer memory 9. The image signals in the divided areas are sequentially read out from the buffer memory 9, and the image signals of the target or scene to be captured and tracked that have been previously stored in the memory 10a as a reference screen are read out, and both are multiplied and The data is input to an adder 11 and roughly scanned at intervals of K/αB _H pixels in the parallel direction and L/βB _V pixels in the vertical direction to calculate a correlation coefficient. Thereafter, it is input to the maximum value detection circuit 12 to find the maximum value of the amplitude of the correlation coefficient for each region, and the region selection circuit 13 is used to select i divided regions in descending order of magnitude.

Next, target acquisition is performed as follows. That is, the detection signal from the area selection circuit 13 described above is input to the area setting circuit 14, and the scanning lines of the i detected areas are multiplied by A _H and A _V in the parallel and perpendicular directions, and the area of each area is determined. After expanding the area, the image signal in this area is stored in the buffer memory 9. The image signals in the divided areas are sequentially read out from the buffer memory 9, and the image signals of the reference screen stored in the memory 10a are read out and both are input to the multiplication and addition circuit 11 to calculate the correlation coefficient. calculate.

After that, the input is input to the maximum value detection circuit 12, the maximum value of the amplitude of the correlation coefficient is determined for each region, and the region selection circuit 13 is used to select divided regions from among these maximum values to give the maximum correlation coefficient. Choose one.

Correlation coefficients are similarly calculated for j correlated images continuously supplied from the imaging device 6 in this divided area, and the maximum value detection circuit 12 calculates the maximum value E ₁ of the amplitude of the correlation coefficient, ..., find E _j .

The j maximum values E ₁ obtained in this way,...
Comparison circuit 1 compares E _j with a preset threshold E _t
5, and after counting the number n _c that exceeds this E _t , a lock-on signal is generated if this n _c exceeds a preset number n _t .

If _nc is smaller than _nt , a lockoff signal is generated and fed back to the area dividing circuit 8 to perform the search operation again.

Next, target tracking is performed as follows. That is, when the lock-on signal is generated from the comparison circuit 15, the tracking window setting circuit 16 is automatically activated.
is activated, and a tracking window is set in an area within the field of view of the imaging device 6 that includes a desired target or scene. An image signal in this area is selected by a switch 17, and either this image signal or the image signal of the reference screen previously stored in the memory 10a is selected by a switch 18 and stored in the memory 10b as a reference screen.

In this case, the reason why the switches 17 and 18 are used to store the data in the memory 10b is to update the reference screen when the target or scene is accompanied by changes such as enlargement, reduction, rotation, etc.

Next, the image signal newly supplied from the imaging device 6 is sent to the buffer memory 9 through the switch 17.
Then, the image signals are read out from the memory 10b and the buffer memory 9 and input to the multiplication and addition circuit 11 to calculate the correlation coefficient.

In this case, the calculation area of the multiplication and addition circuit 11 is limited by a control signal from a correlation calculation control circuit 21, which will be described later.

The two-dimensional correlation coefficient obtained in this manner is input to the tracking point detection circuit 19, and the coordinates giving the maximum value are detected. The distance R between this coordinate signal and the coordinate signal obtained for the image one frame before is calculated by the distance calculation circuit 20 and inputted to the correlation calculation time control circuit 21.

Then, the correlation calculation control circuit 21 compares the distance R with a preset threshold value ε and determines that R≧ε
In this case, a rectangular control signal with a size of 2aR pixels×2aR pixels is generated, and when R<ε, a rectangular control signal with a size of 2ε pixels×2ε pixels is generated.

In this case, the coordinate signal output from the tracking point detection circuit 19 is fed back to the tracking window setting circuit 16 to move the position of the tracking window on the screen, thereby detecting the desired target or scene within the field of view of the imaging device 6. It can be tracked using an electronic loop.

In addition, by inputting the coordinate signal to the tracking control signal generation circuit 22 and outputting an error signal to the servo mount 5 to control the attitude of the imaging device 6, the desired target or scene is kept stationary on the screen and tracked. can do.

Although the above description has been made regarding an image signal supplied from a standard television scanning type imaging device, the present invention is not limited to this, and may be applied to a device that performs correlation processing and tracking of two-dimensional information.

Moreover, although the above description has been made regarding hardware, it goes without saying that the present invention is not limited to this and can be applied to cases where it is realized using software.

Furthermore, although the case where i=3 and j=5 has been described above, the present invention is not limited to this and may be used by setting to any value. Furthermore, although the above description has been made regarding the case where the shape of the control region for correlation calculation is rectangular, the present invention is not limited to this;
Needless to say, it can be applied to any shape such as a fan shape.

As described above, the correlation tracking device according to the present invention can automatically search for and capture a target or scene by correlation processing, and can also perform tracking in real time. is valid. In addition, a tracking system that allows human intervention has the advantage that the operator's labor can be reduced because the operator provides information for determining whether or not to perform tracking.

[Brief explanation of the drawing]

1, 2, and 3 are diagrams for explaining the operation of searching and capturing a target or scene in the correlation tracking device according to the present invention, FIG. 4 is a diagram for explaining the tracking operation, and FIG. 5 is a diagram for explaining the tracking operation. The figure is a configuration diagram of an embodiment of a correlation tracking device according to the present invention. In the figure, 1 is a correlated screen, 2 is a divided area, 3 is a reference screen, 4 is a rectangular correlation calculation area, 5 is a servo mount, 6 is an imaging device, 7 is an analog-digital conversion circuit, and 8 is an area dividing circuit , 9 is a buffer memory, 10a, 10b are memories, 11 is a multiplication and addition circuit, 12 is a maximum value detection circuit, 13 is an area selection circuit, 14 is an area setting circuit, 15 is a comparison circuit, 1
6 is a tracking window setting circuit, 17 and 18 are switches, 19 is a tracking point detection circuit, 20 is a distance calculation circuit, 21 is a correlation calculation control circuit, and 22 is a tracking control signal generation circuit. In the drawings, the same or corresponding parts are denoted by the same reference numerals.

Claims (1)

[Claims] 1. Find the correlation function between the analog-to-digital converted screen of the image signal supplied from the imaging device and the previously stored reference screen, detect the maximum value of the correlation coefficient and the coordinates that give the maximum value, and In a correlation tracking device that tracks a desired target or scene within the field of view, the screen supplied from the image pickup device is
and β, and within each divided area,
an area selection circuit that scans the reference screen two-dimensionally to find the maximum value of the correlation coefficient, and selects i pieces from among the α×β maximum values in descending order; and the i divided areas. an area setting circuit that sets an area that is expanded by A _H and A _V times in parallel and perpendicular directions to the scanning line, and two-dimensionally scans the reference screen on the screen within these areas; A maximum value detection circuit that calculates the correlation coefficient and detects the maximum value of the correlation coefficient for each region, selects one of the divided regions with the largest value among those maximum values, and newly inputs it. For j correlated screens,
Limit the reference screen to the selected area above 2
Scan dimensionally to find the maximum value of the correlation coefficients, count the number n _c of which the maximum value of these j correlation coefficients exceeds a preset threshold, and calculate this counted value n _c
a comparison circuit that determines that a target or scene is larger than n _t (n _t <j); a tracking window setting circuit that sets a tracking window in an area that includes the target or scene determined as described above; A tracking point detection circuit that calculates the correlation coefficient between the screen in this tracking window and the reference screen, and detects the maximum correlation value and the coordinates giving the maximum value as a tracking point, and the coordinates output from this tracking point detection circuit. A correlation tracking device comprising: a tracking control signal generation circuit that outputs an error signal to a servo mount that controls the attitude of the imaging device based on the signal.