JP6687296B2 - Monitoring device - Google Patents

Description

  The present invention relates to a monitoring device that determines whether a target observed by an observation sensor is the same as a target in an image captured by an image sensor.

For example, there is a monitoring device that observes a target object such as an aircraft, a ship, or a vehicle by using a plurality of sensors such as radars or cameras having different observation specifications, and determines the identity of the target observed by the plurality of sensors. is there.
For example, if the monitoring device determines that the target object in the captured image output from the image sensor is the same as the target object observed by a certain radar, the target object observed by the certain radar is detected. The identification information of is superimposed on the captured image and displayed.

An example of a system to which this monitoring device can be applied is an air traffic control support system for observing an aircraft that is a target.
Examples of the sensor included in the monitoring device applied to the air traffic control support system include an airport surface detection radar (ASDE) and an airport surveillance radar (ASR). ASDE can acquire a two-dimensional position by detecting the distance and direction of the target aircraft. The ASR is, for example, a radar that combines a primary surveillance radar (PSR: Primary Surveillance Radar) and a secondary surveillance radar (SSR: Secondary Surveillance Radar). PSR detects the distance and heading of a target aircraft. The SSR detects the distance and direction of the target aircraft by responding to the interrogation signal, which is a radio wave transmitted from the radar antenna, by the transponder mounted on the aircraft, and the response signal includes the aircraft. Since the identification information and the altitude are included, the altitude can be known in addition to the distance and the azimuth.
When the altitude of the aircraft cannot be acquired, these sensors will acquire it as a two-dimensional position, and when the altitude of the aircraft can be acquired, these sensors can acquire it as a three-dimensional position.
In addition, as sensors included in the monitoring device, there are multilateration (MLAT), wide area multilateralization (WAM), ADS-B (Automatic Dependent Surveillance-Broadcast) out / in, and the like.
These sensors can acquire the three-dimensional position of the target aircraft and the identification information of the aircraft.

The monitoring device disclosed in Patent Document 1 below uses an airport surface detection radar as an observation sensor, observes all targets moving on the airport surface, and obtains the positions and identification information of all the targets. I'm trying to get it. Targets include aircraft moving on the airport surface as well as vehicles moving on the airport surface.
In addition, this monitoring device uses a video camera, which is an image sensor, to shoot a target object and acquire a captured image of the target object.
This monitoring device detects the target object in the captured image, by determining the presence or absence of a correlation between the position of the target object on the captured image and the position of the target object acquired by the airport surface detection radar, It is determined whether the target observed by the airport surface detection radar and the target in the image captured by the image sensor are the same.

JP, 11-160424, A

  Since the conventional monitoring device is configured as described above, the identity of the target can be determined if the target is an aircraft or a vehicle moving on the airport surface. However, when the target object is a flying aircraft, it is difficult to accurately determine the three-dimensional position of the target object even if the position coordinates of the target object in the captured image are analyzed. It is impossible to determine whether or not there is a correlation between the three-dimensional position of the object and the three-dimensional position of the target acquired by the airport surface detection radar. Therefore, when the target is an aircraft in flight, it is not possible to determine whether the target observed by the airport surface detection radar and the target in the image captured by the image sensor are the same. There was a problem.

  The present invention has been made to solve the above problems, and even when the target is an aircraft in flight, the target observed by the observation sensor and the target in the image captured by the image sensor It is an object of the present invention to obtain a monitoring device that can determine whether or not the two are the same.

  A monitoring device according to the present invention detects and detects an image sensor that images a region in which a target object exists and outputs a captured image of the region, and a target object in the captured image output from the image sensor. A target detection unit that specifies a target inclusion image area that is a range that includes the target, and the target is observed, the target is observed, and the identification information of the observed target and the target that is the position of the observed target are observed. The size of the target is specified by the observation sensor that outputs the position and the identification information that is output from the observation sensor, and the size of the specified target and the target observation position that is output from the observation sensor Converted to the projection surface, the target inclusion that is the range that includes the target observed by the observation sensor on the projection surface of the image sensor from the size of the converted target and the converted target observation position Observation area And a conversion processing unit for specifying, the first determination processing unit compares the target object inclusion observation area specified by the conversion processing unit with the target object inclusion image area specified by the target detection unit, The target object observed by the observation sensor and the target object detected by the target detection unit are determined to be the same or not.

  According to this invention, the size of the target object is specified by the identification information output from the observation sensor, and the size of the specified target object and the target object observation position output from the observation sensor are displayed on the projection surface of the image sensor. The target inclusion observation region that is the range that includes the target observed by the observation sensor on the projection surface of the image sensor from the converted target size and the converted target observation position. Is provided, and the first determination processing unit compares the target object inclusion observation region specified by the conversion processing unit with the target object inclusion image region specified by the target detection unit and observes Since the target object observed by the sensor and the target object detected by the target detection unit are configured to be determined to be the same or not, even if the target object is a flying aircraft, the observation sensor By And observed target, and the target in the captured image of the image sensor is effective which can be determined whether one and the same.

It is a block diagram which shows the monitoring apparatus by Embodiment 1 of this invention. It is a hardware block diagram which shows the monitoring apparatus by Embodiment 1 of this invention. It is a hardware block diagram of a computer when the components other than the image sensor 1, the observation sensor 3, and the display 16 of the monitoring device are realized by software or firmware. 6 is a flowchart showing a processing procedure in the case where components other than the image sensor 1, the observation sensor 3 and the display 16 of the monitoring device are realized by software or firmware. 5 is an explanatory diagram showing a target object observation position converted to a projection plane of the image sensor 1 by the conversion processing unit 5 and a target object inclusion observation region specified by the conversion processing unit 5. FIG. FIG. 6A is an explanatory diagram showing a first rectangular area corresponding to the entire length and width of the target object, and FIG. 6B is an explanatory diagram showing a second rectangular area corresponding to the entire width and height of the target object. It is explanatory drawing which shows the overlapping range Ov of the target object inclusion observation area and the target object inclusion image area. It is explanatory drawing which shows the overlapping range Ov of the target object inclusion observation area and the 1st and 2nd target object inclusion image area. It is a block diagram which shows the other monitoring apparatus by Embodiment 1 of this invention. 6 is an explanatory diagram showing both inclusion areas specified by an inclusion area specifying unit 8. FIG. It is a block diagram which shows the other monitoring apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the position which weighted and averaged. It is explanatory drawing which shows the positional shift of the target object observation position and the target object image position resulting from the observation sensor 3. FIG. 3 is an explanatory diagram showing an observation error range of a target object observation position and an observation error range on a projection surface of the image sensor 1. It is explanatory drawing which shows the relationship between the eigenvector of an ellipsoid and the width of a long axis. It is explanatory drawing which shows the target inclusion observation region before and after correction, and the observation error range on a projection surface. It is a block diagram which shows the monitoring apparatus by Embodiment 3 of this invention. It is a hardware block diagram which shows the monitoring apparatus by Embodiment 3 of this invention. It is a block diagram which shows the monitoring apparatus by Embodiment 4 of this invention. It is a hardware block diagram which shows the monitoring apparatus by Embodiment 4 of this invention. It is explanatory drawing which shows the projection position calculated by the observation position projection part 61. It is explanatory drawing which shows the relationship between the projection position calculated by the observation position projection part 61, and the target object map coordinate position output from the target detection part 32b. 9 is a flowchart showing the processing content of a second determination processing unit 62. It is a block diagram which shows the monitoring apparatus by Embodiment 5 of this invention. It is a hardware block diagram which shows the monitoring apparatus by Embodiment 5 of this invention. It is explanatory drawing which shows the reference position of image sensors 1-1 to 1-N and the angle of image sensors 1-1 to 1-N. It is a block diagram which shows the monitoring apparatus by Embodiment 6 of this invention.

  Hereinafter, in order to describe the present invention in more detail, embodiments for carrying out the present invention will be described with reference to the accompanying drawings.

Embodiment 1.
1 is a block diagram showing a monitoring device according to a first embodiment of the present invention. FIG. 2 is a hardware configuration diagram showing the monitoring device according to the first embodiment of the present invention.
1 and 2, the image sensor 1 is realized by, for example, an optical camera or an infrared camera.
The image sensor 1 images, for example, a region inside or around an airport site where a moving body is traffic as a region in which a target object exists, and a captured image of the region is acquired by the target detection unit 2 and the display processing unit 7. Output to. The targets include not only aircraft but also vehicles traveling on the ground. Further, if the airport is near the sea, ships and the like are included, and if there is a railroad, rail cars and the like are included.

The target detection unit 2 is realized by, for example, the target detection circuit 11 shown in FIG.
The target detection unit 2 executes a process of detecting a target object in the captured image output from the image sensor 1.
The target detection unit 2 first determines each of the target object image position that is the position of the detected target object on the captured image and the target object inclusion image region that is the range that includes the target object on the captured image. Processing for outputting to the processing unit 6 and the display processing unit 7 is performed.

The observation sensor 3 is a sensor other than the image sensor 1, and is realized by, for example, ASR, ASDE, MLAT, WAM, ADS-Bout / in, or the like.
The observation sensor 3 observes the target, and outputs the identification information of the observed target and the target observation position, which is a three-dimensional position of the target, to the conversion processing unit 5. The identification information of the target includes, for example, the flight number of the aircraft and a unique identification number of the target such as the body of the aircraft.
The object shape information storage unit 4 is realized by, for example, the shape storage circuit 12 shown in FIG.
The object shape information storage unit 4 stores shape information indicating the shapes of various target objects. For example, as the shape information corresponding to the identification information of the target, the total length, width, height, etc. of the target indicated by the identification information are stored.

The conversion processing unit 5 is realized by, for example, the conversion processing circuit 13 shown in FIG.
The conversion processing unit 5 reads the shape information corresponding to the identification information output from the observation sensor 3 from the object shape information storage unit 4 and refers to the shape information to determine the size of the target observed by the observation sensor 3. The process of identifying the height is performed.
The conversion processing unit 5 performs a process of converting each of the size of the specified target object and the target object observation position output from the observation sensor 3 into a projection plane of the image sensor 1.
The conversion processing unit 5 is a range that includes the target object observed by the observation sensor 3 on the projection surface of the image sensor 1 from the converted size of the target object and the converted target object observation position. Perform processing to specify the object inclusion observation area.
In the first embodiment, the conversion processing unit 5 identifies the size of the target observed by the observation sensor 3 by referring to the shape information corresponding to the identification information output from the observation sensor 3, Although the example in which the size of the target is converted to the projection surface of the image sensor 1 is shown, the present invention is not limited to this.
For example, when it is difficult for the observation sensor 3 to obtain the identification information of the target object, the conversion processing unit 5 converts the preset size of the target object into the projection surface of the image sensor 1. May be.
The size of the target object set in advance may be, for example, the size on the catalog.

The first determination processing unit 6 is realized by, for example, the first determination processing circuit 14 shown in FIG.
The first determination processing unit 6 compares the target object inclusion observation region specified by the conversion processing unit 5 with the target object inclusion image region output from the target detection unit 2 and is observed by the observation sensor 3. A process of determining whether or not the target object and the target object detected by the target detection unit 2 are the same is performed.
That is, the first determination processing unit 6 divides the target object inclusion observation region specified by the conversion processing unit 5 and the target object inclusion image region output from the target detection unit 2 on the projection surface of the image sensor 1. A process of calculating the overlapping range is performed.
In addition, the first determination processing unit 6 determines that the target object observed by the observation sensor 3 and the target object detected by the target detection unit 2 are the same object if the overlapping range is equal to or more than the threshold value. If the overlapping range is less than the threshold value, a process of determining that the target object observed by the observation sensor 3 and the target object detected by the target detection unit 2 are not the same object is performed.

The display processing unit 7 is realized by, for example, the display processing circuit 15 shown in FIG.
The display processing unit 7 performs a process of displaying the captured image output from the image sensor 1 on the display 16.
When the first determination processing unit 6 determines that they are the same object, the display processing unit 7 outputs the target object observation position converted to the projection surface of the image sensor 1 by the conversion processing unit 5 and the observation sensor 3. Each of the identification information is displayed on the captured image, and the target object inclusion image area output from the target detection unit 2 is displayed on the captured image.
Even if the first determination processing unit 6 does not determine that they are the same object, the display processing unit 7 causes the conversion processing unit 5 to convert the target object observation position and the observation sensor to the projection surface of the image sensor 1. 3 of the identification information, the target inclusion observation region specified by the conversion processing unit 5, the target object image position output from the target detection unit 2, and the target inclusion image region output from the target detection unit 2. May be independently displayed on the captured image.
The display 16 is, for example, a display device such as a liquid crystal display.

  In FIG. 1, the image sensor 1, the target detection unit 2, the observation sensor 3, the object shape information storage unit 4, the conversion processing unit 5, the first determination processing unit 6, the display processing unit 7, and the display, which are the constituent elements of the monitoring device. Each of 16 is assumed to be realized by dedicated hardware as shown in FIG. That is, what is realized by the image sensor 1, the target detection circuit 11, the observation sensor 3, the shape memory circuit 12, the conversion processing circuit 13, the first determination processing circuit 14, the display processing circuit 15, and the display 16 is assumed. .

Here, the shape memory circuit 12 is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Accessible Memory). Alternatively, a volatile semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like is applicable.
The target detection circuit 11, the conversion processing circuit 13, the first determination processing circuit 14, and the display processing circuit 15 are, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, and an ASIC (Application Specific). An integrated circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof is applicable.

The components other than the image sensor 1, the observation sensor 3, and the display 16 of the monitoring device are not limited to those realized by dedicated hardware, but are realized by software, firmware, or a combination of software and firmware. It may be one.
Software or firmware is stored in the memory of the computer as a program. The computer means hardware that executes a program, and includes, for example, a CPU (Central Processing Unit), a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, a DSP (Digital Signal Processor), and a GPU (Graphics). Processing Unit) is applicable.

FIG. 3 is a hardware configuration diagram of a computer when the components other than the image sensor 1, the observation sensor 3, and the display 16 of the monitoring device are realized by software or firmware.
When the components other than the image sensor 1, the observation sensor 3, and the display 16 of the monitoring device are realized by software or firmware, the object shape information storage unit 4 is configured on the memory 21 of the computer and the target detection unit 2, A program for causing a computer to execute the processing procedures of the object shape information storage unit 4, the conversion processing unit 5, the first determination processing unit 6, and the display processing unit 7 is stored in the memory 21, and the processor 22 of the computer stores the program in the memory 21. It suffices to execute the stored program.

FIG. 4 is a flowchart showing a processing procedure when the components of the monitoring device except the image sensor 1, the observation sensor 3 and the display 16 are realized by software or firmware.
Further, FIG. 2 shows an example in which each component of the monitoring device is realized by dedicated hardware, and in FIG. 3, the components other than the image sensor 1, the observation sensor 3 and the display 16 of the monitoring device are software and Although an example in which it is implemented by firmware is shown, some of the components of the monitoring device may be implemented by dedicated hardware, and the remaining components may be implemented by software or firmware.

Next, the operation will be described.
The image sensor 1 images, for example, a region inside or around an airport site where a moving body is traffic as a region in which a target object exists, and a captured image of the region is acquired by the target detection unit 2 and the display processing unit 7. Output to. The targets include not only aircraft but also vehicles traveling on the ground.

The target detection unit 2 performs a process of detecting a target object in the captured image output from the image sensor 1 (step ST1 in FIG. 4).
Since the process itself for detecting the target object in the captured image is a known technique, detailed description thereof will be omitted.
The target detection unit 2 outputs the detected target image position, which is the position of the detected target on the captured image, to the first determination processing unit 6 and the display processing unit 7.
In addition, the target detection unit 2 sets the first determination processing unit 6 and a target object inclusion image area that is a range that includes the target object in the captured image and an ID (IDentification) that identifies the detected target object. It is output to the display processing unit 7. This ID is a number or the like arbitrarily assigned by the target detection unit 2.
Here, the target inclusion image area corresponds to, for example, a rectangular frame that surrounds the target on the captured image, or a frame configured by a curved line and a straight line.
Further, the target object image position corresponds to, for example, the position coordinates of the pixel existing in the center of the target object inclusion image region.

In the first embodiment, it is assumed that the monitoring device has one image sensor 1, but the monitoring device may have a plurality of image sensors 1.
When the monitoring device is equipped with a plurality of image sensors 1, the target detection unit 2 may detect the target object independently for each image sensor 1, or the target images may be captured by the plurality of image sensors 1. The captured image may be combined into one captured image as a panoramic image, and the target object may be detected on the combined captured image. Further, the plurality of image sensors 1 independently detect the target object and at the same time operate in cooperation with the other image sensor 1, so that the same target object existing across the plurality of image sensors 1 is the same. The target object having the same ID may be prevented from being detected by another image sensor 1 or output to the processing unit in the subsequent stage.

The observation sensor 3 observes the target object, and converts the identification information of the observed target object and the target object observation position (X1, Y1, Z1), which is the three-dimensional position of the target object, into the conversion processing unit 5 and the display processing unit. Output to 7. Further, as described above, when handling a single observation sensor 3, depending on the sensor, only the distance and direction as in ASR and only the two-dimensional position on the ground surface as in ASDE may be acquired. In the case of ASR, the atmospheric altitude of the aircraft or the atmospheric altitude is corrected to the standard atmosphere and converted to the altitude from the sea surface, and the distance, bearing, and altitude are converted to a three-dimensional position (X1, Y1, Z1) and output. It is okay to do it. In the case of ASDE, it is possible to set a default value (a value that is realistically possible under the rules of air traffic control) as the altitude of the target and output the three-dimensional position (X1, Y1, Z1). To do.
The identification information of the target includes a unique identification number of the target, such as the flight number of the aircraft or the information of the aircraft body.
Further, the observation sensor 3 may output the velocity vector indicating the estimated velocity of the target to the conversion processing unit 5 if the velocity vector can be acquired.

As shown in FIG. 5, the conversion processing unit 5 projects the target object observation position (X1, Y1, Z1) output from the observation sensor 3 onto the projection surface of the image sensor 1 so that the target object observation position (X1 , Y1, Z1) is converted into coordinates (u, v) on the projection surface of the image sensor 1 (step ST2 in FIG. 4).
FIG. 5 is an explanatory diagram showing the target object observation position converted to the projection surface of the image sensor 1 by the conversion processing unit 5 and the target object inclusion observation region specified by the conversion processing unit 5.
The coordinates (u, v) on the projection surface of the image sensor 1 correspond to the coordinates of the central pixel in the target inclusion observation region.

式（１）において、Ａは、画像センサ１の内部パラメータ、Ｒは、回転行列、Ｔは、平行移動ベクトルである。
Ｘ_{ｗｏｒｌｄ}，Ｙ_{ｗｏｒｌｄ}，Ｚ_{ｗｏｒｌｄ}は、目標物が存在している３次元直交座標系の各軸の座標値である。例えば、観測センサ３が設置されている位置を原点とする北基準直交座標系が考えられる。
ｕ_{ｉｍａｇｅ}，ｖ_{ｉｍａｇｅ}は、画像センサ１の投影面上での各軸の座標値である。
したがって、目標物観測位置（Ｘ１，Ｙ１，Ｚ１）を式（１）の座標値（Ｘ_{ｗｏｒｌｄ}，Ｙ_{ｗｏｒｌｄ}，Ｚ_{ｗｏｒｌｄ}）に代入すると、式（１）の画像センサ１の投影面上での座標値（ｕ_{ｉｍａｇｅ}，ｖ_{ｉｍａｇｅ}）が、画像センサ１の投影面の座標（ｕ，ｖ）となる。For example, the conversion processing unit 5 can convert the target object observation position (X1, Y1, Z1) into the coordinates (u, v) on the projection surface of the image sensor 1 by using the following formula (1). .

In Expression (1), A is an internal parameter of the image sensor 1, R is a rotation matrix, and T is a translation vector.
X _world , Y _world , and Z _world are the coordinate values of each axis of the three-dimensional rectangular coordinate system in which the target exists. For example, a north reference Cartesian coordinate system whose origin is the position where the observation sensor 3 is installed can be considered.
u _image and v _image are coordinate values of each axis on the projection surface of the image sensor 1.
Therefore, when the target object observation position (X1, Y1, Z1) is substituted into the coordinate values (X _world , Y _world , Z _world ) of the equation (1), the coordinates on the projection surface of the image sensor 1 of the equation (1) are calculated. The value (u _image , v _image ) is the coordinate (u, v) on the projection surface of the image sensor 1.

Upon receiving the identification information of the target object from the observation sensor 3, the conversion processing unit 5 reads out the shape information corresponding to the identification information from the object shape information storage unit 4.
The conversion processing unit 5 grasps the total length, width, and height of the target by referring to the shape information. When the identification information of the target is not input, for example, general length, width, and height may be set in advance as default values and used.
Further, when the conversion processing unit 5 receives the velocity vector from the observation sensor 3, the conversion processing unit 5 grasps the traveling direction of the target object from the velocity vector. When the velocity vector is not input, for example, according to the rules of air traffic control, a default value of the velocity vector may be set according to a preset course or a general velocity to grasp the traveling direction of the target object. I will.
FIG. 6 is an explanatory diagram showing the target object inclusion observation region specified by the conversion processing unit 5.
FIG. 6A shows a first rectangular area corresponding to the entire length and width of the target object, and FIG. 6B shows a second rectangular area corresponding to the entire width and height of the target object.

In the conversion processing unit 5, as shown in FIG. 6A, the length of the side corresponding to the moving direction of the target is the total length of the target, and the length of the side corresponding to the direction orthogonal to the moving direction is the length of the target. Generate a first rectangular area that is the full width.
Further, in the conversion processing unit 5, as shown in FIG. 6B, the length of the side corresponding to the height direction is the height of the target object, and the length of the side corresponding to the horizontal direction is the total width of the target object. A second rectangular area is generated.
Here, even when the traveling direction of the target object cannot be grasped, the rectangular area may be generated based on the preset traveling direction and inclination of the target object. On the other hand, if detailed information such as the pitch, yaw, and roll of the target object can be obtained, it may be used.

Next, as shown in FIG. 6A, the conversion processing unit 5 selects the diagonal line D1 of the two diagonal lines D1 and D2 in the first rectangular area, the diagonal line having a direction close to a right angle with the line-of-sight direction of the image sensor 1. To do.
The conversion processing unit 5 projects the endpoints a1 and a2 of the selected diagonal line D1 onto the projection surface of the image sensor 1 to transform the endpoints a1 and a2 of the diagonal line D1 into coordinates a3 and a4 on the projection surface of the image sensor 1. .

In order to convert the end points a1 and a2 of the diagonal line D1 into the coordinates a3 and a4 of the projection surface of the image sensor 1, it is necessary to obtain the coordinates of the three-dimensional orthogonal coordinate system at the end points a1 and a2 of the diagonal line D1.
If the target object observation position (X1, Y1, Z1) output from the observation sensor 3 corresponds to the center position of the first rectangular area, the target object observation position (X1, Y1, Z1) and the first position From the size of the rectangular area, the coordinates of the three-dimensional orthogonal coordinate system at the end points a1 and a2 of the diagonal line D1 can be obtained.
However, in general, the target object observation position (X1, Y1, Z1) output from the observation sensor 3 does not always correspond to the center position of the first rectangular area.
When the target object observation position (X1, Y1, Z1) does not correspond to the center position of the first rectangular area, it is difficult to obtain the coordinates of the three-dimensional orthogonal coordinate system at the end points a1 and a2 of the diagonal line D1. ..
Therefore, in the first embodiment, for example, the end points a1 and a2 of the diagonal line D1 are converted into the coordinates a3 and a4 of the projection surface of the image sensor 1 as follows.

Since the end points a1 and a2 of the diagonal line D1 are end points of the diagonal line D1 in the first rectangular area 9, the distance between the end point a1 and the end point a2 can be calculated. Therefore, for example, the coordinates of the three-dimensional Cartesian coordinate system at the end point a2 can be represented by the coordinates of the three-dimensional Cartesian coordinate system at the end point a1.
By substituting the unknown coordinates of the end point a1 into the equation (1), the coordinates of the end point a1 are converted into the coordinates a3 of the projection surface of the image sensor 1, and the coordinates of the end point a2 represented by the coordinates of the end point a1 are represented by the expression (1 ), The coordinate of the end point a2 is converted into the coordinate a4 of the projection surface of the image sensor 1, so that the distance L1-2 between the coordinate a3 and the coordinate a4 can be obtained.
Since the coordinates (u, v), which is the target observation position on the projection surface of the image sensor 1, have already been calculated, the coordinates of the projection surface of the image sensor 1 are calculated from the coordinates (u, v) and the distance L1-2. It is possible to calculate a3 and a4.

Further, as shown in FIG. 6B, the conversion processing unit 5 selects the diagonal line D3 of the two diagonal lines D3 and D4 in the second rectangular region, which is in a direction close to a right angle with the line-of-sight direction of the image sensor 1. .
The conversion processing unit 5 projects the endpoints b1 and b2 of the selected diagonal line D3 onto the projection surface of the image sensor 1 to transform the endpoints b1 and b2 of the diagonal line D3 into coordinates b3 and b4 of the projection surface of the image sensor 1. .
The process of converting the end points b1 and b2 of the diagonal line D3 into the coordinates b3 and b4 of the projection surface of the image sensor 1 is similar to the process of converting the end points b1 and b2 of the diagonal line D1 into the coordinates b3 and b4 of the projection surface of the image sensor 1. Can be done.

From the coordinates a3, a4 and the coordinates b3, b4 of the converted projection plane, the conversion processing unit 5 determines, on the projection surface of the image sensor 1, a target object that is a range including the target object, as shown in FIG. The inclusive observation area is specified (step ST3 in FIG. 4).
In FIG. 5, the coordinates of the upper left end point of the target inclusion observation area are (a3, b3), and the coordinates of the lower left end point of the target inclusion observation area are (a3, b4).
The coordinates of the upper right end point of the target inclusion observation area are (a4, b3), and the coordinates of the lower right end point of the target inclusion observation area are (a4, b4).

The first determination processing unit 6 compares the target object inclusion observation region specified by the conversion processing unit 5 with the target object inclusion image region output from the target detection unit 2 and is observed by the observation sensor 3. It is determined whether or not the target object and the target object detected by the target detection unit 2 are the same object.
That is, as shown in FIG. 7, the first determination processing unit 6 outputs the target inclusion observation region specified by the conversion processing unit 5 and the target detection unit 2 on the projection surface of the image sensor 1. An overlapping range Ov with the target object inclusion image area is calculated (step ST4 in FIG. 4).
FIG. 7 is an explanatory diagram showing an overlapping range Ov between the target-subject inclusion observation area and the target-subject inclusion image area.

As shown in FIG. 7, the first determination processing unit 6 compares the calculated overlapping range Ov with a preset threshold Th (step ST5 in FIG. 4).
If the overlapping range Ov is greater than or equal to the threshold Th (step ST5: YES in FIG. 4), the first determination processing unit 6 detects the target object observed by the observation sensor 3 and the target detection unit 2. It is determined that the target object is the same object (step ST6 in FIG. 4).
If the overlapping range Ov is less than the threshold Th (step ST5: NO in FIG. 4), the first determination processing unit 6 detects the target observed by the observation sensor 3 and the target detection unit 2. It is determined that the target and the target are not the same (step ST7 in FIG. 4).

The display processing unit 7 displays the captured image output from the image sensor 1 on the display 16.
When the first determination processing unit 6 determines that they are the same object, the display processing unit 7 outputs the target object observation position converted to the projection surface of the image sensor 1 by the conversion processing unit 5 and the observation sensor 3. Each of the identification information is displayed on the captured image, and the target object inclusion image area output from the target detection unit 2 is displayed on the captured image (step ST8 in FIG. 4).

When the first determination processing unit 6 determines that they are not the same object, the display processing unit 7 captures an image of the target object image position, the target object inclusion image region, and the target object ID output from the target detection unit 2. It is displayed above (step ST9 in FIG. 4).
Alternatively, when the first determination processing unit 6 determines that they are not the same object, the display processing unit 7 converts the identification information output from the observation sensor 3 and the conversion processing unit 5 into the projection surface of the image sensor 1. The target object observation position and the target object inclusion observation area specified by the conversion processing unit 5 are displayed on the captured image.

  As is clear from the above, according to the first embodiment, the size of the target is specified by the identification information output from the observation sensor 3, and the size of the specified target and the output from the observation sensor 3 are output. Each of the target object observation positions was converted to the projection surface of the image sensor 1, and the observation sensor 3 observed on the projection surface of the image sensor 1 from the size of the converted target object and the converted target object observation position. A conversion processing unit 5 that specifies a target object inclusion observation region that is a range that includes a target object is provided, and the first determination processing unit 6 causes the target object inclusion observation region specified by the conversion processing unit 5 and the target. It is determined whether the target object observed by the observation sensor 3 and the target object detected by the target detection unit 2 are the same object by comparing the target object inclusion image area specified by the detection unit 2 with each other. It is configured to do. Thereby, even if the target is an aircraft in flight, it is possible to determine whether the target observed by the observation sensor 3 and the target in the captured image of the image sensor 1 are the same. There is an effect that can be done.

In the first embodiment, an example in which one target object is detected on the captured image by the target detection unit 2 is shown. However, for example, the front portion and the rear portion of one target object are separated and May be detected as one target.
In such a case, the target detection unit 2, as shown in FIG. 8, includes a target object inclusion image area (hereinafter, referred to as a first target object inclusion image area) which is a range including a front portion of one target object. The target object-containing image area (hereinafter, referred to as a second target object-containing image area) that is a range including the rear portion of one target object is specified.
FIG. 8 is an explanatory diagram showing an overlapping range Ov between the target object inclusion observation area and the first and second target object inclusion image areas.

The first determination processing unit 6 calculates, on the projection surface of the image sensor 1, an overlapping range Ov1 between the first target inclusion image area and the target inclusion observation area specified by the conversion processing unit 5, Further, the overlapping range Ov2 between the target object inclusion observation area specified by the conversion processing unit 5 and the second target object inclusion image area is calculated.
If the sum of the overlapping range Ov1 and the overlapping range Ov2 is greater than or equal to the threshold value Th, the first determination processing unit 6 determines that the plurality of target objects detected by the target detecting unit 2 are part of one target object, It is determined that the target observed by the sensor 3 and one target are the same.
If the total of the overlapping range Ov1 and the overlapping range Ov2 is less than the threshold Th, the first determination processing unit 6 determines that the target observed by the observation sensor 3 and one target are not the same.

In the first embodiment, an example in which the display processing unit 7 displays the target object inclusion image area output from the target detection unit 2 or the target object inclusion observation area specified by the conversion processing unit 5 on the captured image. Is shown.
This is merely an example, and for example, the display processing unit 7 may display both the inclusion area, which is an area including both the target inclusion image area and the target inclusion observation area, on the captured image.
FIG. 9 is a configuration diagram showing another monitoring device according to the first embodiment of the present invention.
In FIG. 9, the inclusion area specifying unit 8 is realized by, for example, an inclusion area specifying circuit.
As shown in FIG. 10, the inclusion area specifying unit 8 includes the target object inclusion observation area specified by the conversion processing unit 5 and the target object inclusion image output from the target detecting unit 2 on one projection plane of the image sensor. A process of specifying a both-inclusive region, which is a region including both the region and the region, is performed.
FIG. 10 is an explanatory diagram showing both inclusion areas specified by the inclusion area specifying unit 8.
The display processing unit 7 displays the both inclusion areas specified by the inclusion area specifying unit 8 on the captured image output from the image sensor 1.

In the first embodiment, the display processing unit 7 captures an image of the target object observation position converted to the projection surface of the image sensor 1 by the conversion processing unit 5 or the target object image position output from the target detection unit 2. The example displayed above is shown.
This is merely an example, and for example, the display processing unit 7 may perform a weighted average of the target object observation position and the target object image position and display the weighted average position on the captured image.
FIG. 11 is a configuration diagram showing another monitoring device according to the first embodiment of the present invention.
In FIG. 11, the average position output unit 9 is realized by, for example, an average position output circuit.
As shown in FIG. 12, the average position output unit 9 weights the target object observation position converted to the projection surface of the image sensor 1 by the conversion processing unit 5 and the target object image position output from the target detection unit 2. A process of averaging and outputting the weighted average position is executed.
FIG. 12 is an explanatory diagram showing the weighted average positions.
The display processing unit 7 displays the position output from the average position output unit 9 on the captured image output from the image sensor 1.

Embodiment 2.
The above-described first embodiment shows an example in which the conversion processing unit 5 specifies the target object inclusion observation region.
In the second embodiment, an example will be described in which the conversion processing unit 5 corrects the target object inclusion observation region according to the observation error information indicating the observation error range of the target object observation position.
The configuration diagram of the monitoring apparatus according to the second embodiment is FIG. 1, FIG. 9 or FIG. 11 similarly to the monitoring apparatus according to the first embodiment.

The position of the target observed by the observation sensor 3 includes an observation error. Therefore, as shown in FIG. 13, the target object observation position output from the observation sensor 3 and the target object image position output from the target detection unit 2 may deviate significantly.
FIG. 13 is an explanatory diagram showing the positional deviation between the target object observation position and the target object image position due to the observation sensor 3.

In the second embodiment, the observation sensor 3 outputs a three-dimensional observation error covariance matrix to the conversion processing unit 5 as the observation error information indicating the observation error range of the target object observation position.
As shown in FIG. 14, the conversion processing unit 5 projects the observation error range of the target object observation position indicated by the observation error information output from the observation sensor 3 onto the projection surface of the image sensor 1 to obtain the target object observation position. The observation error range of is converted into coordinates on the projection surface of the image sensor 1.
FIG. 14 is an explanatory diagram showing the observation error range of the target object observation position and the observation error range on the projection surface of the image sensor 1.
The observation error range of the target object observation position is geometrically represented by an ellipsoid as shown in FIG.
The conversion processing unit 5 substitutes the position of each point in the observation error range of the target object observation position into the coordinate values (X _world , Y _world , Z _world ) of the equation (1) to observe the target object observation position. The error range can be converted into the coordinates of the projection surface of the image sensor 1.
The following processing may be used as the processing for converting the observation error range of the target object observation position into the coordinates of the projection surface of the image sensor 1.

The conversion processing unit 5 calculates the width of the major axis and the width of the minor axis of the ellipsoid from the eigenvalue and the eigenvector of the ellipsoid, which are the observation error ranges of the target observation position.
FIG. 15 is an explanatory diagram showing the relationship between the eigenvector of the ellipsoid and the width of the major axis.
Then, the conversion processing unit 5 converts each of the long-axis width end points and the short-axis width end points of the ellipsoid into the coordinates of the projection surface of the image sensor 1 using the formula (1), thereby performing projection. Calculate the observation error range on the surface.

As shown in FIG. 16, when the observation error range on the projection surface does not include the target inclusion observation area specified by the same method as in the above-described first embodiment, the conversion processing unit 5 displays it on the projection surface. The observation error range on the projection plane is expanded so that the observation error range in (3) includes the target inclusion observation region.
Next, the conversion processing unit 5 corrects the target-included observation area such that the target-included observation area matches the expanded observation error range.
That is, as shown in FIG. 16, the conversion processing unit 5 sets the enlarged observation error range as the corrected target object inclusion observation area.
The conversion processing unit 5 outputs the corrected target object inclusion observation region to the first determination processing unit 6.
FIG. 16 is an explanatory diagram showing a target inclusion observation region before and after correction and an observation error range on the projection surface.
Since the processing content of the first determination processing unit 6 is the same as that in the first embodiment, detailed description thereof will be omitted.

  As is clear from the above, according to the second embodiment, the observation sensor 3 outputs the observation error information indicating the observation error range of the target object observation position, and the conversion processing unit 5 outputs the observation sensor 3 from the observation sensor 3. Since the target inclusion observation region is corrected according to the observation error information, even if the observation error of the observation sensor 3 is large, it is possible to prevent deterioration of the determination accuracy of the same object.

Embodiment 3.
In the third embodiment, a first tracking processing unit is provided between the target detection unit 2 and the first determination processing unit 6, and a second tracking processing unit is provided between the observation sensor 3 and the conversion processing unit 5. An example provided with will be described.

FIG. 17 is a configuration diagram showing a monitoring device according to Embodiment 3 of the present invention. FIG. 18 is a hardware configuration diagram showing a monitoring device according to the third embodiment of the present invention.
In FIGS. 17 and 18, the same reference numerals as those in FIGS. 1 and 2 indicate the same or corresponding portions, and thus the description thereof will be omitted.
The image sensors 1-1 to 1-N are realized by, for example, an optical camera or an infrared camera, like the image sensor 1 in FIG.
The image sensors 1-1 to 1-N image, for example, an area inside or around an airport site where a mobile body is in traffic as an area in which a target object exists, and a captured image of the area is generated by a video generation unit. 31 and the target detection unit 32.

The video generation unit 31 is realized by, for example, the video generation circuit 41 shown in FIG.
The video generation unit 31 outputs each of the captured images output from the image sensors 1-1 to 1-N to the display processing unit 40.
In addition, the video generation unit 31 generates one panoramic image by combining the captured images output from the image sensors 1-1 to 1-N, and the panoramic image is used as the captured image, and the target detection unit 32 and the display process are performed. It is output to the unit 40. Here, the captured images output from the image sensors 1-1 to 1-N may be separately output directly to the target detection unit 32 and the display processing unit 40 as captured images.

The target detection unit 32 is realized by, for example, the target detection circuit 42 shown in FIG.
The target detection unit 32 is a target in the captured image output from the image sensor 1-n (n = 1, 2, ..., N) or a target in the captured image output from the video generation unit 31. A process for detecting an object is performed.
The target detection unit 32 performs a first tracking process on each of the target object image position which is the position of the target object detected on the captured image and each of the target object inclusion image region which is the range including the target object on the captured image. Processing for outputting to the unit 33 is performed.

The first tracking processing unit 33 is realized by, for example, the first tracking processing circuit 43 shown in FIG.
The first tracking processing unit 33 performs the tracking process of the target object detected by the target detection unit 32, and sets the target object inclusion image area, which is a range including the target object after the tracking process, to the first target image area. Processing for outputting to the determination processing unit 37 is performed.
As the target tracking process executed by the first tracking processing unit 33, for example, a known tracking process using a Kalman filter, a particle filter, or the like can be used.

The observation sensors 3-1 to 3-M are realized by, for example, ASR, MLAT, WAM or the like, like the observation sensor 3 in FIG. 1.
The observation sensor 3-m (m = 1, 2, ..., M) observes the target object, and outputs the identification information of the observed target object and the target object observation position which is the three-dimensional position of the target object. Output to the second tracking processing unit 34-m and the multi-sensor tracking processing unit 35.

The second tracking processing unit 34-m is realized by, for example, the second tracking processing circuit 44 shown in FIG.
The second tracking processing unit 34-m performs the tracking process of the target object using the target object observation position output from the observation sensor 3-m, and observes the target object which is the position of the target object after the tracking process. A process of outputting the position to the conversion processing unit 36 is performed.
In addition, the second tracking processing unit 34-m performs a process of outputting the identification information of the target output from the observation sensor 3-m to the conversion processing unit 36.
The multi-sensor tracking processing unit 35 is realized by, for example, the multi-sensor tracking processing circuit 45 shown in FIG.
The multi-sensor tracking processing unit 35 performs tracking processing of the target object using the target object observation positions output from the M observation sensors 3-1 to 3-M, and at the position of the target object after the tracking processing. A process of outputting a certain target object observation position to the conversion processing unit 36 is performed.
The multi-sensor tracking processing unit 35 differs from the second tracking processing unit 34-m only in that a plurality of target object observation positions are used.
As the target tracking process performed by each of the second tracking processing unit 34-m and the multi-sensor tracking processing unit 35, for example, a known tracking process using a Kalman filter, a particle filter, or the like can be used.

The conversion processing unit 36 is realized by, for example, the conversion processing circuit 46 illustrated in FIG.
The conversion processing unit 36 reads the shape information corresponding to the identification information output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35 from the object shape information storage unit 4 and refers to the shape information. Then, the process of specifying the size of the target observed by the observation sensor 3-m is performed.
The conversion processing unit 36 sets the size of the identified target object and each of the target object observation positions output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35 onto the projection surface of the image sensor 1-n. Perform the conversion process.
The conversion processing unit 36 includes the target object observed by the observation sensor 3-m on the projection plane of the image sensor 1-n from the converted size of the target object and the converted target object observation position. A process of specifying a target inclusion observation region that is a range is performed.

The first determination processing unit 37 is realized by, for example, the first determination processing circuit 47 shown in FIG.
The first determination processing unit 37 compares the target object inclusion observation area specified by the conversion processing unit 36 with the target object inclusion image area output from the first tracking processing unit 33, and the observation sensor 3-. A process of determining whether or not the target observed by m and the target detected by the target detector 32 are the same is performed.
That is, the first determination processing unit 37, on the projection surface of the image sensor 1-n, the target inclusion observation region specified by the conversion processing unit 36 and the target object output from the first tracking processing unit 33. A process of calculating the overlapping range with the included image area is performed.
In addition, the first determination processing unit 37 determines that the target object observed by the observation sensor 3-m and the target object detected by the target detection unit 32 are the same object if the overlapping range is equal to or more than the threshold value. Perform the determination process.
If the overlapping range is less than the threshold, the first determination processing unit 37 determines that the target object observed by the observation sensor 3-m and the target object detected by the target detection unit 32 are not the same object. Carry out.
The determination result storage unit 38 is realized by, for example, the determination result storage circuit 48 shown in FIG.
The determination result storage unit 38 stores the determination result of the first determination processing unit 37.

The smoothing processing unit 39 is realized by, for example, the smoothing processing circuit 49 shown in FIG.
The smoothing processing unit 39 performs processing for smoothing the target object inclusion image area output from the first tracking processing unit 33 in the time direction and outputting the smoothed target object inclusion image area to the display processing unit 40. .
The display processing unit 40 is realized by, for example, the display processing circuit 50 shown in FIG.
The display processing unit 40 performs a process of displaying the captured image output from the video generation unit 31 on the display 16.
When the first determination processing unit 37 determines that the display processing unit 40 is the same object, the conversion processing unit 36 converts the target object observation position and the observation sensor 3-converted to the projection surface of the image sensor 1-n. Each of the identification information output from m is displayed on the captured image, and the smoothed target object inclusion image area output from the smoothing processing unit 39 is displayed on the captured image.

  In FIG. 17, the image sensor 1-n, the observation sensor 3-m, the image generation unit 31, the target detection unit 32, the first tracking processing unit 33, and the second tracking processing unit 34-m, which are the components of the monitoring device. 18, the multi-sensor tracking processing unit 35, the conversion processing unit 36, the first determination processing unit 37, the determination result storage unit 38, the smoothing processing unit 39, the display processing unit 40, and the display 16 are dedicated as shown in FIG. It is supposed to be realized by the hardware of. That is, the image sensor 1-n, the observation sensor 3-m, the image generation circuit 41, the target detection circuit 42, the first tracking processing circuit 43, the second tracking processing circuit 44, the multi-sensor tracking processing circuit 45, and the conversion processing circuit. 46, the first determination processing circuit 47, the determination result storage circuit 48, the smoothing processing circuit 49, the display processing circuit 50, and the display 16 are assumed.

Here, the shape storage circuit 12 and the determination result storage circuit 48 are, for example, a nonvolatile or volatile semiconductor memory such as a RAM, a ROM, a flash memory, an EPROM, an EEPROM, a magnetic disk, a flexible disk, an optical disk, a compact disk, This corresponds to mini discs, DVDs, etc.
Further, the image generation circuit 41, the target detection circuit 42, the first tracking processing circuit 43, the second tracking processing circuit 44, the multi-sensor tracking processing circuit 45, the conversion processing circuit 46, the first determination processing circuit 47, and the smoothing processing. The circuit 49 and the display processing circuit 50 are, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

  The components other than the image sensor 1-n, the observation sensor 3-m, and the display 16 of the monitoring device are not limited to those realized by dedicated hardware, but software, firmware, or a combination of software and firmware. May be realized by.

  When the components other than the image sensor 1-n, the observation sensor 3-m, and the display 16 of the monitoring device are realized by software or firmware, the object shape information storage unit 4 and the determination result storage unit 38 are shown in FIG. Of the image generation unit 31, the target detection unit 32, the first tracking processing unit 33, the second tracking processing unit 34-m, the multi-sensor tracking processing unit 35, the conversion processing unit 36, and the second processing unit 36. A program for causing a computer to execute the processing procedure of the determination processing unit 37, the smoothing processing unit 39, and the display processing unit 40 of No. 1 is stored in the memory 21, and the processor 22 of the computer executes the program stored in the memory 21. You can do it like this.

Next, the operation will be described.
The image sensor 1-n (n = 1, 2, ..., N) images, for example, an area within the airport premises where the mobile body is trafficing and an area around the airport as an area in which the target object exists, The captured image of the area is output to the target detection unit 32 and the video generation unit 31.
The video generation unit 31 outputs the captured image output from the image sensor 1-n to the display processing unit 40.
Further, the video generation unit 31 generates one panoramic image by combining the N captured images output from the image sensors 1-1 to 1-N, and the target detection unit 32 uses the panoramic image as the captured image. And output to the display processing unit 40.

The target detection unit 32 performs a process of detecting a target object in the captured image output from the image sensor 1-n or a target object in the captured image output from the video generation unit 31.
Since the process itself for detecting the target object in the captured image is a known technique, detailed description thereof will be omitted.
In the third embodiment, for simplification of description, it is assumed that the target detection unit 32 detects one target.
The target detection unit 32 outputs the target object image position, which is the position of the target object detected on the captured image, and the ID for identifying the detected target object, to the first tracking processing unit 33 and the display processing unit 40.
The target detection unit 32 also outputs a target object inclusion image area, which is a range including the target object detected on the captured image, to the first tracking processing unit 33.

The first tracking processing unit 33 uses the target object image position output from the target detection unit 32 to perform the tracking process of the target object in the captured image, and the range including the target object after the tracking process. The target inclusion image area, which is, is output to the first determination processing unit 37 and smoothing processing unit 39.
The target tracking process by the first tracking processing unit 33 is a process of obtaining the correlation of the target object image position output from the target detection unit 32 in the time series direction, and for example, a Kalman filter can be used.
The first tracking processing unit 33 may smooth the target object inclusion image area and output the smoothed target object inclusion image area to the first determination processing unit 37. By smoothing the target object inclusion image area, the fluctuation of the target object inclusion image area can be suppressed, so that the determination accuracy of the first determination processing unit 37 can be improved.

The observation sensor 3-m (m = 1, 2, ..., M) observes the target object, and identifies the observed target object identification information and the target object observation position, which is the three-dimensional position of the target object, as the second position. Output to the tracking processing unit 34-m and the multi-sensor tracking processing unit 35.
Further, the observation sensor 3-m outputs a velocity vector indicating the estimated velocity of the target to the second tracking processing unit 34-m.
In the third embodiment, for simplification of description, it is assumed that one target is observed by the observation sensor 3-m.

The second tracking processing unit 34-m performs tracking processing of the target object observed by the observation sensor 3-m by using the target object observation position and velocity vector output from the observation sensor 3-m, and performs tracking. The target observation position, which is the position of the processed target, is output to the conversion processing unit 36.
The target tracking process performed by the second tracking processing unit 34-m is a process of taking a correlation in the time series direction of the target observation position output from the observation sensor 3-m, and for example, a Kalman filter can be used. .
The second tracking processing unit 34-m also outputs the identification information of the target output from the observation sensor 3-m, the estimated error covariance matrix calculated during the tracking processing, and the like to the conversion processing unit 36.
The estimation error covariance matrix corresponds to estimation error information indicating the estimation error due to the target tracking process.

The multi-sensor tracking processing unit 35 uses the target observation position and velocity vector output from the M observation sensors 3-1 to 3-M to perform tracking processing of the target observed by the observation sensor 3-m. The target object observation position, which is the position of the target object after the tracking process, is output to the conversion processing unit 36.
Further, the multi-sensor tracking processing unit 35 outputs the estimation error covariance matrix calculated during the tracking processing to the conversion processing unit 36.
By the multi-sensor tracking processing unit 35 performing the tracking process of the target, even when any of the M observation sensors 3-1 to 3-M cannot acquire the identification information of the target, The multi-sensor tracking processing unit 35 can associate the identification information acquired by another observation sensor with the target object after the tracking processing.
Therefore, the multi-sensor tracking processing unit 35 determines that the target object after the tracking processing is performed even if there is an observation sensor that cannot acquire the identification information of the target in the M observation sensors 3-1 to 3-M. Can be output to the conversion processing unit 36.

The conversion processing unit 36 reads the shape information corresponding to the identification information output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35 from the object shape information storage unit 4 and refers to the shape information. Then, the size of the target observed by the observation sensor 3-m is specified.
The conversion processing unit 36 sets the size of the identified target object and each of the target object observation positions output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35 onto the projection surface of the image sensor 1-n. Perform the conversion process.
The conversion processing unit 36 includes the target object observed by the observation sensor 3-m on the projection plane of the image sensor 1-n from the converted size of the target object and the converted target object observation position. The target inclusion observation area that is the range is specified.
The processing content of the conversion processing unit 36 is generally the same as the processing content of the conversion processing unit 5 in the second embodiment, except for the following points.

The conversion processing unit 5 in the second embodiment corrects the target object inclusion observation region in accordance with the observation error information indicating the observation error range of the target object observation position.
On the other hand, the conversion processing unit 36 according to the third embodiment replaces the observation error range of the target object observation position with the tracking output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. The target inclusion observation region is corrected according to the estimation error covariance matrix showing the estimation error due to the processing.
The correction process itself of the target inclusion observation region by the conversion processing unit 36 is the same as the correction process of the target inclusion observation region by the conversion processing unit 5 and thus detailed description thereof will be omitted. Since the range is narrower than the observation error range of the target object observation position, it is possible to further improve the determination accuracy of the same object as compared with the second embodiment.

The first determination processing unit 37 compares the target object inclusion observation area corrected by the conversion processing unit 36 with the target object inclusion image area output from the first tracking processing unit 33, and the observation sensor 3-. It is determined whether the target object observed by m and the target object detected by the target detection unit 32 are the same object.
That is, the first determination processing unit 37, on the projection surface of the image sensor 1-n, the target inclusion observation region corrected by the conversion processing unit 36, and the target object output from the first tracking processing unit 33. An overlapping range Ov with the included image area is calculated.

The first determination processing unit 37 compares the calculated overlapping range Ov with a preset threshold Th.
If the overlapping range Ov is equal to or greater than the threshold Th, the first determination processing unit 37 determines that the target object observed by the observation sensor 3-m and the target object detected by the target detection unit 32 are the same object. judge.
If the overlapping range Ov is less than the threshold Th, the first determination processing unit 37 determines that the target object observed by the observation sensor 3 and the target object detected by the target detection unit 32 are not the same object.

When the processing timing of the determination processing by the first determination processing unit 37 corresponds to the frame rate of the image sensor 1-n, the processing load of the first determination processing unit 37 becomes large, and the processing load of the image sensor 1-n increases. The determination process may not end at the frame rate.
Therefore, the first determination processing unit 37 stores the determination result of whether or not they are the same in the determination result storage unit 38, and in the determination result storage unit 38 without performing the determination process for a certain period. The stored determination result may be read and the determination result may be output to the display processing unit 40.
When the first determination processing unit 37 determines that the target object observed by the observation sensor 3 and the target object detected by the target detection unit 32 are the same object, the target object ID, the target object The determination result storage unit 38 stores information such as the identification information, the target inclusion image area, and the target inclusion observation area.
Then, the first determination processing unit 37 may output the information stored in the determination result storage unit 38 to the display processing unit 40 together with the determination result.

式（２）において、Ｌｋは、時刻ｔｋにおける目標物包含画像領域の横幅（または縦幅）、Ｌｓ，ｋ−１は、時刻ｔｋ−１における目標物包含画像領域の横幅推定値（または縦幅推定値）、Ｌｓ，ｋは、時刻ｔｋにおける目標物包含画像領域の横幅推定値（または縦幅推定値）、ａは係数である。
平滑処理部３９が目標物包含画像領域を時間方向に平滑化することで、目標物包含画像領域の横幅又は縦幅の変動を抑えることができる。The smoothing processing unit 39 smoothes the target object inclusion image area output from the first tracking processing unit 33 in the time direction as shown in the following Expression (2), and the smoothed target object inclusion image area is obtained. It is output to the display processing unit 40.

In Expression (2), Lk is the width (or height) of the target-included image area at time tk, and Ls, k−1 is the estimated width (or height) of the target-included image area at time tk-1. Estimated value), Ls, k are horizontal width estimated values (or vertical width estimated values) of the target inclusion image area at time tk, and a is a coefficient.
The smoothing processing unit 39 smoothes the target object-containing image area in the time direction, so that it is possible to suppress variation in the horizontal width or the vertical width of the target object-containing image area.

The display processing unit 40 displays the captured image of the image sensor 1-n output from the video generation unit 31 or the captured image that is a panoramic image on the display 16.
When the first determination processing unit 37 determines that the display processing unit 40 is the same object, the display processing unit 40 converts the target object observation position and the observation sensor 3-m converted to the projection plane of the image sensor 1 by the conversion processing unit 36. Each of the output identification information is displayed on the captured image, and the smoothed target object inclusion image area output from the smoothing processing unit 39 is displayed on the captured image.

When the first determination processing unit 37 determines that the objects are not the same, the display processing unit 40 outputs the target object image position output from the target detection unit 32 and the smoothed target object output from the smoothing processing unit 39. The inclusion image area and the target object ID output from the target detection unit 32 are displayed on the captured image.
Alternatively, when the first determination processing unit 37 determines that they are not the same object, the display processing unit 7 outputs the identification information output from the observation sensor 3-m and the conversion processing unit 36 to project the image sensor 1-n. The target object observation position converted into the surface and the target object inclusion observation region corrected by the conversion processing unit 36 are displayed on the captured image.

In the third embodiment, the first determination processing unit 37 compares the target object inclusion observation area corrected by the conversion processing unit 36 with the target object inclusion image area output from the first tracking processing unit 33. An example is shown.
However, since the update rate of the observation sensor 3-m is lower than that of the image sensor 1-n, a time difference occurs between the observation time of the observation sensor 3-m and the imaging time of the image sensor 1-n. Sometimes. The time difference described above becomes a cause of deterioration of the determination accuracy of the first determination processing unit 37.
Even if the observation time of the observation sensor 3-m and the imaging time of the image sensor 1-n are the same, the data delay time between the observation sensor 3-m and the first determination processing unit 37 and the image A time difference may occur between the sensor 1-n and the data delay time between the first determination processing unit 37. The time difference described above becomes a cause of deterioration of the determination accuracy of the first determination processing unit 37.

Therefore, the first determination processing unit 37 may eliminate the deterioration factor of the determination accuracy as follows.
First, the first determination processing unit 37 sets the update time t corresponding to the update rate of the image sensor 1-n, for example.
When the observation time of the observation sensor 3-m is behind the update time t, the first determination processing unit 37 outputs the update time t and the target object observation position from the second tracking processing unit 34-m. Or a time difference (t−t ′) from the time t ′ at which the target object observation position is output from the multi-sensor tracking processing unit 35.

式（４）において、Ｉｎ×ｎは、ｎ行ｎ列の単位行列、Ｏｎ×ｎは、ｎ行ｎ列の零行列である。Next, the first determination processing unit 37 is calculated by the time difference (t−t ′), for example, by the tracking process of the second tracking processing unit 34-m as shown in the following Expression (3). The estimated state vector x hat t is extrapolated over time. In the text of the specification, since the symbol "^" cannot be added on the character x due to the electronic application, it is expressed as "x hat t". The estimated state vector x hat t corresponds to the target object observation position at time t after the tracking processing by the second tracking processing unit 34-m or the like.
Φ (t−t ′) in equation (3) can be defined as a state transition matrix of a constant velocity linear motion model, for example, as shown in equation (4) below.

In Equation (4), In × n is an identity matrix of n rows and n columns, and On × n is a zero matrix of n rows and n columns.

式（５）において、Ｑ（ｔ−ｔ’）は、駆動雑音共分散行列である。
第１の判定処理部３７が、時間外挿した推定状態ベクトルｘハットｔである追尾処理後の目標物観測位置及び時間外挿した推定誤差共分散行列Ｐｔを変換処理部３６に出力することで、時刻差（ｔ−ｔ’）による判定精度の劣化要因が解消される。Further, the first determination processing unit 37 uses the time difference (t−t ′) to calculate the estimation error due to the tracking process of the second tracking processing unit 34-m as shown in the following Expression (5). The estimated error covariance matrix Pt shown is extrapolated over time.

In Expression (5), Q (t−t ′) is a driving noise covariance matrix.
The first determination processing unit 37 outputs the time-extrapolated estimated state vector x hat t, which is the target observation position after the tracking processing, and the time-extrapolated estimation error covariance matrix Pt, to the conversion processing unit 36. , The deterioration factor of the determination accuracy due to the time difference (t−t ′) is eliminated.

  As is clear from the above, according to the third embodiment, the tracking process of the target detected by the target detection unit 32 is performed, and the target that is the range including the target after the tracking process is included. The position of the target after the tracking processing is performed by performing the tracking processing of the target observed by the observation sensor 3-m and the first tracking processing section 33 that outputs the inclusion image area to the first determination processing section 37. And a second tracking processing unit 34-m that outputs the target object observation position to the conversion processing unit 36. This suppresses the imaging error of the target by the image sensor 1-n and the observation error of the target by the observation sensor 3-m. As a result, it is possible to further improve the determination accuracy of the same object as compared with the first embodiment.

Fourth Embodiment
In the above-described first to third embodiments, the example in which the monitoring device includes the first determination processing unit 6 or the first determination processing unit 37 is shown.
In the fourth embodiment, an example will be described in which the monitoring device includes a first determination processing unit 37, a second determination processing unit 62, and a third determination processing unit 64.
Further, unlike Embodiments 1 to 3, Embodiment 4 is characterized in that the same object is determined by using the position in map coordinates instead of on the image.

FIG. 19 is a configuration diagram showing a monitoring device according to Embodiment 4 of the present invention. FIG. 20 is a hardware configuration diagram showing a monitoring device according to Embodiment 4 of the present invention.
19 and 20, the same reference numerals as those in FIGS. 1, 2, 17, and 18 indicate the same or corresponding portions, and thus the description thereof will be omitted.
The observation position projection unit 61 is realized by, for example, the observation position projection circuit 71 shown in FIG.
The observation position projection unit 61 sets the target object observation position output from the second tracking processing unit 34-m or the target object observation position output from the multi-sensor tracking processing unit 35 in the line-of-sight direction of the image sensor 1-n. When moving, a process of calculating a position intersecting the ground plane as a projected position is performed.

The second determination processing unit 62 is realized by, for example, the second determination processing circuit 72 shown in FIG.
The second determination processing unit 62 compares the projection position calculated by the observation position projection unit 61 with the target object map coordinate position output from the target detection unit 32b, and the target observed by the observation sensor 3-m. A process of determining whether the object and the target detected by the target detection unit 32b are the same is performed.
Here, the target detection unit 32b carries out a process of detecting a target object, similarly to the target detection unit 32 in FIG.
However, as a method for the target detection unit 32b to calculate the target map coordinate position, the target pixel position on the image is compared with the coordinate position on the map using a map DB prepared in advance. But good. On the other hand, since the target in flight is not a target traveling on the ground, it is difficult to collate it with a map. Therefore, as shown in FIG. May be output as the target map coordinate position.

The third tracking processing unit 63 is realized by, for example, the third tracking processing circuit 73 shown in FIG.
The third tracking processing unit 63, among the one or more target objects detected by the target detection unit 32b, tracks a target object other than the target object determined to be the same object by the second determination processing unit 62. Processing is performed to output the position of the target object after the tracking processing.

The third determination processing unit 64 is realized by, for example, the third determination processing circuit 74 shown in FIG.
The third determination processing unit 64 includes the target observation position output from the second tracking processing unit 34-m or the target observation position output from the multi-sensor tracking processing unit 35, and the third tracking processing unit 63. Whether the target observed by the observation sensor 3-m and the target detected by the target detector 32b are the same by comparing the position of the target after the tracking processing output from The process for determining is executed.
The target selection unit 65 is realized by, for example, the target selection circuit 75 shown in FIG.
The target selection unit 65 selects one or more targets from the targets determined to be the same by the first determination processing unit 37, the second determination processing unit 62, or the third determination processing unit 64. A process of selecting an item is performed.

In the monitoring device of FIG. 19, the observation position projection unit 61, the second determination processing unit 62, the third tracking processing unit 63, the third determination processing unit 64, and the target selection unit 65 are applied to the monitoring device of FIG. However, it may be applied to the monitoring device of FIG. 1.
In FIG. 19, the image sensor 1-n, the observation sensor 3-m, the image generation unit 31, the target detection unit 32b, the first tracking processing unit 33, and the second tracking processing unit 34-m that are the constituent elements of the monitoring device. , Multi-sensor tracking processing unit 35, conversion processing unit 36, first determination processing unit 37, determination result storage unit 38, smoothing processing unit 39, display processing unit 40, observation position projection unit 61, second determination processing unit 62. It is assumed that each of the third tracking processing unit 63, the third determination processing unit 64, the target selecting unit 65, and the display 16 is realized by dedicated hardware as shown in FIG. That is, the image sensor 1-n, the observation sensor 3-m, the image generation circuit 41, the target detection circuit 42, the first tracking processing circuit 43, the second tracking processing circuit 44, the multi-sensor tracking processing circuit 45, and the conversion processing circuit. 46, the first determination processing circuit 47, the determination result storage circuit 48, the smoothing processing circuit 49, the display processing circuit 50, the observation position projection circuit 71, the second determination processing circuit 72, the third tracking processing circuit 73, the third It is assumed that the determination processing circuit 74, the target selection circuit 75, and the display 16 of FIG.

  Image generation circuit 41, target detection circuit 42, first tracking processing circuit 43, second tracking processing circuit 44, multi-sensor tracking processing circuit 45, conversion processing circuit 46, first determination processing circuit 47, smoothing processing circuit 49. The display processing circuit 50, the observation position projection circuit 71, the second determination processing circuit 72, the third tracking processing circuit 73, the third determination processing circuit 74, and the target selection circuit 75 are, for example, a single circuit, a composite circuit. A circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination of these.

The components other than the image sensor 1-n, the observation sensor 3-m, and the display 16 of the monitoring device are not limited to those realized by dedicated hardware, but software, firmware, or a combination of software and firmware. May be realized by.
When the components other than the image sensor 1-n, the observation sensor 3-m, and the display 16 of the monitoring device are realized by software or firmware, the object shape information storage unit 4 and the determination result storage unit 38 are stored in the computer shown in FIG. Of the image generation unit 31, the target detection unit 32b, the first tracking processing unit 33, the second tracking processing unit 34-m, the multi-sensor tracking processing unit 35, the conversion processing unit 36, and the second processing unit 36. 1 determination processing unit 37, smoothing processing unit 39, display processing unit 40, observation position projection unit 61, second determination processing unit 62, third tracking processing unit 63, third determination processing unit 64, and target selection. A program for causing a computer to execute the processing procedure of the unit 65 is stored in the memory 21, and the processor 22 of the computer executes the program stored in the memory 21. It may be set to cormorants.

Next, the operation will be described.
As shown in FIG. 21, the observation position projection unit 61 uses the image sensor to detect the target object observation position output from the second tracking processing unit 34-m or the target object observation position output from the multi-sensor tracking processing unit 35. A position that intersects the ground plane when moving in the line-of-sight direction 1-n is calculated as the projective position.
FIG. 21 is an explanatory diagram showing projection positions calculated by the observation position projection unit 61.
FIG. 22 is an explanatory diagram showing the relationship between the projection position calculated by the observation position projection unit 61 and the target object map coordinate position output from the target detection unit 32b.
If the target observed by the observation sensor 3-m and the target on the captured image, which is the target detected by the target detector 32b, are the same, the projection position and the target shown in FIG. The distance from the map coordinate position is short, but if they are not the same object, the distance between the projection position and the target map coordinate position is long.

The second determination processing unit 62 compares the projection position calculated by the observation position projection unit 61 with the target object map coordinate position output from the target detection unit 32b and is observed by the observation sensor 3-m. It is determined whether the target object and the target object detected by the target detection unit 32b are the same object.
FIG. 23 is a flowchart showing the processing contents of the second determination processing unit 62.
Hereinafter, the processing content of the second determination processing unit 62 will be specifically described with reference to FIG.
In the fourth embodiment, it is assumed that the target detection unit 32b detects I target objects and the observation sensor 3-m observes J target objects.

When the second determination processing unit 62 receives the target object observation position of the target object j (j = 1, 2, ..., J) from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. , The altitude Z of the target j is recognized.
The second determination processing unit 62 compares the altitude Z of the target object j with a preset threshold value Zth (step ST11 in FIG. 23).
If the altitude Z of the target j is equal to or higher than the threshold Zth (step ST11: YES in FIG. 23), the second determination processing unit 62 recognizes that the target j is a moving body in flight (FIG. 23, step ST12).
If the altitude Z of the target object j is less than the threshold value Zth (step ST11: NO in FIG. 23), the second determination processing unit 62 recognizes that the target object j is a moving object existing on the ground. (Step ST13 of FIG. 23). The moving body existing on the ground includes a moving body that is stopped.

When the second determination processing unit 62 recognizes that the target object j is a moving body in flight, the target object j that is a moving body in flight and the target object i (i = , 1, ..., I) are the same or not.
Specifically, the second determination processing unit 62 acquires the projection position of the target object j calculated by the observation position projection unit 61 (step ST14 in FIG. 23).

式（６）において、
ｘハット _{ｔ，ｊ，ＨＥＴ} は、観測位置射影部６１により算出された目標物ｊの時刻ｔにおける射影位置を示す推定位置ベクトル
ｘハット _{ｔ，ｊ，ＣＡＭ} は、目標検出部３２ｂにより検出された目標物ｉの時刻ｔにおける目標物地図座標位置を示す推定位置ベクトル
Ｐ _{ｔ，ｊ，ＨＥＴ} は、第２の追尾処理部３４−ｍ又はマルチセンサ追尾処理部３５の追尾処理による目標物ｊの時刻ｔにおける推定位置誤差を示す推定誤差共分散行列
Ｐ _{ｔ，ｉ，ＣＡＭ} は、目標検出部３２ｂにより検出された目標物ｉの時刻ｔにおける推定位置誤差を示す推定誤差共分散行列 Next, the second determination processing unit 62 outputs the projection position of the target object j calculated by the observation position projection unit 61 and the target detection unit 32b, for example, as shown in the following equation (6). Using the target object map coordinate position of the target object i, the test value ε of the chi-square test is calculated (step ST15 in FIG. 23).

In equation (6),
The x-hat _{t, j, HET} is an estimated position vector indicating the projection position of the target object j calculated by the observation position projection unit 61 at time t. The x-hat _{t, j, CAM} is the target detected by the target detection unit 32b. The estimated position vector P _{t, j, HET} indicating the target map coordinate position of the object i at the time t is the time t of the target _{j by} the tracking processing of the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. The estimation error covariance matrix P _{t, i, CAM} indicating the estimated position error at is the estimation error covariance matrix indicating the estimated position error at time t of the target object i detected by the target detection unit 32b.

閾値εｔｈは、所定の有意水準に基づいて、例えば、カイ二乗分布表から求めたものである。
例えば、有意水準を５％として検定すると、危険率５％で、本来同一である航跡が、同一でない航跡と誤った判定がなされることを意味する。
第２の判定処理部６２は、カイ二乗検定の検定値εが閾値εｔｈ未満であれば（図２３のステップＳＴ１６：ＹＥＳの場合）、飛行中の移動体である目標物ｊと、目標検出部３２ｂにより検出された目標物ｉとが同一物であると判定する（図２３のステップＳＴ１７）。
第２の判定処理部６２は、カイ二乗検定の検定値εが閾値εｔｈ以上であれば（図２３のステップＳＴ１６：ＮＯの場合）、飛行中の移動体である目標物ｊと、目標検出部３２ｂにより検出された目標物ｉとが同一物でないと判定する（図２３のステップＳＴ１８）。
第２の判定処理部６２は、目標物ｉと同一物であると判定した目標物ｊの識別情報を第３の判定処理部６４及び目標物選択部６５に出力し、目標物ｊと同一物であると判定した目標物ｉのＩＤを第３の追尾処理部６３に出力する。
第２の判定処理部６２による飛行中の移動体である目標物ｊと、目標検出部３２ｂにより検出された目標物ｉとの同一物か否かの判定処理は、飛行中の移動体である目標物ｊと目標検出部３２ｂにより検出された目標物ｉとの全ての組み合わせについて行われる。Next, the second determination processing unit 62 compares the test value ε of the chi-square test with the preset threshold εth as shown in the following equation (7) (step ST16 in FIG. 23).

The threshold value εth is obtained from, for example, a chi-square distribution table based on a predetermined significance level.
For example, if the significance level is 5%, it means that the originally identical tracks are erroneously determined as non-identical tracks at a risk rate of 5%.
If the test value ε of the chi-square test is less than the threshold value εth (step ST16 of FIG. 23: YES), the second determination processing unit 62 and the target j that is a moving body in flight and the target detection unit. It is determined that the target object i detected by 32b is the same object (step ST17 in FIG. 23).
If the test value ε of the chi-square test is greater than or equal to the threshold value εth (step ST16: NO in FIG. 23), the second determination processing unit 62 and the target object j that is a moving body in flight and the target detection unit. It is determined that the target object i detected by 32b is not the same object (step ST18 in FIG. 23).
The second determination processing unit 62 outputs the identification information of the target object j that is determined to be the same as the target object i to the third determination processing unit 64 and the target object selection unit 65, and the same object as the target object j. The ID of the target object i determined to be is output to the third tracking processing unit 63.
The second determination processing unit 62 determines whether or not the target object j, which is a moving object in flight, and the target object i detected by the target detection unit 32b are the same object. This is performed for all combinations of the target object j and the target object i detected by the target detection unit 32b.

The third tracking processing unit 63 compares the IDs of the I target objects output from the target detecting unit 32b with the IDs of the target objects i output from the second determination processing unit 62, and calculates the I A target f having an ID different from the ID of the target i is selected from the targets.
The third tracking processing unit 63 uses the target object map coordinate position for the selected target object f from the target object map coordinate positions of the I target objects output from the target detection unit 32b to determine the target object f. The tracking process of is executed.
The third tracking processing unit 63 outputs the target object map coordinate position , which is the position of the target object f after the tracking processing, to the third determination processing unit 64.
The tracking process of the target object f by the third tracking processing unit 63 is a process of obtaining the correlation of the target object map coordinate position of the selected target object f in the time series direction, and for example, a Kalman filter can be used.

The third determination processing unit 64 outputs the identification information of J target objects output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35, and the second determination processing unit 62. The identification information of the target object j is compared, and a target object g having identification information different from the identification information of the target object j is selected from J target objects.
The third determination processing unit 64 selects the target of the selected target g from the target observation positions of the J target objects output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. Select the object observation position.
The third determination processing unit 64 determines the target object observation position of the selected target object g and the target object map coordinate position that is the position of the target object f after the tracking processing output from the third tracking processing unit 63. By comparison, it is determined whether the target object g observed by the observation sensor 3-m and the target object f detected by the target detection unit 32b are the same object.
Hereinafter, the processing content of the third determination processing unit 64 will be specifically described.

式（８）において、
ｘハット _{ｔ，ｇ，ＨＥＴ} は、第２の追尾処理部３４−ｍ又はマルチセンサ追尾処理部３５から出力された目標物ｇの時刻ｔにおける目標物観測位置及び速度を含む推定状態ベクトル
ｘハット _{ｔ，ｆ，ＣＡＭ} は、第３の追尾処理部６３から出力された追尾処理後の時刻ｔにおける目標物ｆの位置及び速度を含む推定状態ベクトル
Ｐ _{ｔ，ｇ，ＨＥＴ} は、第２の追尾処理部３４−ｍ又はマルチセンサ追尾処理部３５の追尾処理による目標物ｇの時刻ｔにおける推定位置誤差を示す推定誤差共分散行列
Ｐ _{ｔ，ｆ，ＣＡＭ} は、第３の追尾処理部６３の追尾処理による目標物ｆの時刻ｔにおける推定位置誤差を示す推定誤差共分散行列 The third determination processing unit 64, for example, as shown in the following equation (8), the target object observation position of the target object g, and the target object f after the tracking process output from the third tracking processing unit 63. And the position of are used to calculate the test value ε of the chi-square test.

In equation (8),
x-hat _{t, g, HET} is an estimated state vector x-hat _t including the target object observation position and velocity at time t of the target object g output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. _{, F, CAM} is an estimated state vector P _{t, g, HET} including the position and velocity of the target f at the time t after the tracking processing output from the third tracking processing unit 63 is the second tracking processing unit. 34-m or the estimation error covariance matrix P _{t, f, CAM} indicating the estimated position error of the target g at the time t by the tracking processing of the multi-sensor tracking processing unit 35 is obtained by the tracking processing of the third tracking processing unit 63. An estimation error covariance matrix showing the estimated position error of the target f at time t

第３の判定処理部６４は、カイ二乗検定の検定値εが閾値εｔｈ未満であれば、観測センサ３−ｍにより観測された目標物ｇと、目標検出部３２ｂにより検出された目標物ｆとが同一物であると判定する。
第３の判定処理部６４は、カイ二乗検定の検定値εが閾値εｔｈ以上であれば、観測センサ３−ｍにより観測された目標物ｇと、目標検出部３２ｂにより検出された目標物ｆとが同一物でないと判定する。
第３の判定処理部６４は、目標物ｆと同一物であると判定した目標物ｇの識別情報を目標物選択部６５に出力する。Next, the third determination processing unit 64 compares the test value ε of the chi-square test with the preset threshold εth as shown in the following expression (9).

If the test value ε of the chi-square test is less than the threshold εth, the third determination processing unit 64 determines the target g observed by the observation sensor 3-m and the target f detected by the target detector 32b. Are determined to be the same.
If the test value ε of the chi-square test is greater than or equal to the threshold value εth, the third determination processing unit 64 determines the target object g observed by the observation sensor 3-m and the target object f detected by the target detection unit 32b. Are not the same.
The third determination processing unit 64 outputs the identification information of the target object g that is determined to be the same as the target object f to the target object selecting unit 65.

The target selection unit 65 selects one or more targets from among the targets determined to be the same by the first determination processing unit 37, the second determination processing unit 62, or the third determination processing unit 64. Select an item.
For example, the target selection unit 65 selects any target from the targets indicated by the identification information output from the second determination processing unit 62.
Next, the target selection unit 65 has identification information different from the identification information output from the second determination processing unit 62 from among the targets indicated by the identification information output from the third determination processing unit 64. Select a target.
Next, the target selection unit 65 selects the identification information output from the second determination processing unit 62 and the third determination processing from among the targets indicated by the identification information output from the first determination processing unit 37. A target object having identification information different from the identification information output from the unit 64 is selected.
When the target is selected, the target selecting unit 65 outputs the identification information of the selected target, the target observation position and the target observation position of the selected target to the display processing unit 40.

The display processing unit 40 displays the captured image of the image sensor 1-n output from the video generation unit 31 or the captured image that is a panoramic image on the display 16.
When the first determination processing unit 37 determines that they are the same object, the display processing unit 40 includes the target object observation position output from the target object selection unit 65 and the identification information output from the observation sensor 3-m. Each of them is displayed on the captured image, and the smoothed target object inclusion image area output from the smoothing processing unit 39 is displayed on the captured image.

When the first determination processing unit 37 determines that the objects are not the same object, the display processing unit 40 outputs the target object map coordinate position output from the target detection unit 32b and the smoothed target output from the smoothing processing unit 39. The object inclusion image area and the target object ID output from the target detection unit 32b are displayed on the captured image.
Alternatively, when the first determination processing unit 37 determines that the objects are not the same object, the display processing unit 7 outputs the identification information output from the observation sensor 3-m and the target object observation output from the target object selection unit 65. The position and the target inclusion observation region are displayed on the captured image.

  As is clear from the above, according to the fourth embodiment, the target object observation position output from the second tracking processing unit 34-m or the target object observation position output from the multi-sensor tracking processing unit 35 is From the observation position projection unit 61 that calculates a position intersecting the ground plane as a projection position when moving in the line-of-sight direction of the image sensor 1-n, and the projection position calculated by the observation position projection unit 61 and the target detection unit 32b. A second method of comparing the output target observation position and determining whether the target observed by the observation sensor 3-m and the target detected by the target detector 32b are the same. And the determination processing unit 62. As a result, it is possible to further improve the determination accuracy of the same object as compared with the first to third embodiments.

  Further, according to the fourth embodiment, among the one or more target objects detected by the target detection unit 32b, target objects other than the target objects determined to be the same by the second determination processing unit 62. Of the target object observation position, which is the position of the target object after the tracking process, and the target object observation position output from the observation sensor 3-m. 3 is compared with the target object observation position output from the tracking processing unit 63 of No. 3, and whether the target object observed by the observation sensor 3-m and the target object detected by the target detection unit 32b are the same object. The third determination processing unit 64 for determining whether or not it is provided. As a result, it is possible to further improve the determination accuracy of the same object as compared with the first to third embodiments.

Embodiment 5.
In the fifth embodiment, the first conversion processing unit 82 converts the target object map coordinate position into an angle with the reference position of the image sensor 1-n as the origin, and the second conversion processing unit 85 causes the target object to be converted. An example will be described in which the observation position is converted into an angle with the reference position of the image sensor 1-n as the origin, and the determination processing unit 86 compares the conversion angles of both.

24 is a configuration diagram showing a monitoring device according to a fifth embodiment of the present invention. FIG. 25 is a hardware configuration diagram showing a monitoring device according to the fifth embodiment of the present invention.
24 and 25, the same reference numerals as those in FIGS. 1, 2, 17, 18, 19, and 20 denote the same or corresponding portions, and thus description thereof will be omitted.
The first correspondence information storage unit 81 is realized by, for example, the first correspondence information storage circuit 91 shown in FIG.
The first correspondence information storage unit 81 stores the azimuth angle and the elevation angle, which are angles with the reference position of the image sensors 1-1 to 1-N as the origin, and the target object map coordinate position output from the target detection unit 32. It is a database that stores correspondence relationships.
When the number of image sensors 1 is one, the first correspondence information storage unit 81 associates the azimuth angle and the elevation angle with the installation position of one image sensor 1 as the origin with the target map coordinate position. It may be one that stores relationships.

The first conversion processing unit 82 is realized by, for example, the first conversion processing circuit 92 shown in FIG.
The first conversion processing unit 82 converts the target object map coordinate position output from the target detection unit 32 into an azimuth angle and an elevation angle that are angles with the reference position of the image sensors 1-1 to 1-N as the origin. Carry out.
That is, the first conversion processing unit 82 refers to the correspondence relationship stored in the first correspondence information storage unit 81, and the azimuth angle and the azimuth angle corresponding to the target object map coordinate position output from the target detection unit 32. Perform the process to obtain the elevation angle.

The first tracking processing unit 83 is realized by, for example, the first tracking processing circuit 93 shown in FIG.
The first tracking processing unit 83 performs tracking processing of the azimuth angle and the elevation angle converted by the first conversion processing unit 82, and the azimuth angle and the elevation angle after the tracking processing and the angular velocity of the azimuth angle after the tracking processing. And the angular velocity of the elevation angle are output.
The tracking processing of the azimuth angle and the elevation angle by the first tracking processing unit 83 is a process of obtaining the time series direction correlation of the azimuth angle and the elevation angle converted by the first conversion processing unit 82, and for example, a Kalman filter is used. You can

The second correspondence information storage section 84 is realized by, for example, the second correspondence information storage circuit 94 shown in FIG.
The second correspondence information storage unit 84 stores the azimuth angle and the elevation angle, which are angles with the reference position of the image sensors 1-1 to 1-N as the origin, and the second tracking processing unit 34-m or the multi-sensor tracking processing unit. 35 is a database that stores the correspondence relationship between the target observation position and the target observation speed output from 35.
When the number of the image sensors 1 is one, the second correspondence information storage unit 84 causes the azimuth angle and the elevation angle whose origin is the installation position of one image sensor 1, the target object observation position, and the target object observation. It may store the correspondence with the speed.

The second conversion processing unit 85 is realized by, for example, the second conversion processing circuit 95 shown in FIG.
The second conversion processing unit 85 uses the target object observation position output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35 as the reference position of the image sensors 1-1 to 1-N as the origin. A process of converting the azimuth angle and the elevation angle, which are angles, is performed.
That is, the second conversion processing unit 85 refers to the correspondence relationship stored in the second correspondence information storage unit 84, and is output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. The process of acquiring the azimuth angle and the elevation angle corresponding to the target observation position is performed.
In addition, the second conversion processing unit 85 sets the target observation speed output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35 to the origin of the reference positions of the image sensors 1-1 to 1-N. The processing of converting into the angular velocity of the azimuth angle and the angular velocity of the elevation angle is performed.
That is, the second conversion processing unit 85 refers to the correspondence relationship stored in the second correspondence information storage unit 84, and is output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. The processing for acquiring the azimuth angular velocity and the elevation angular velocity corresponding to the target observation speed is performed.

The determination processing unit 86 is realized by, for example, the determination processing circuit 96 shown in FIG.
The determination processing unit 86 includes the azimuth angle, the elevation angle, the angular velocity of the azimuth angle, and the angular velocity of the elevation angle after the tracking processing, which are output from the first tracking processing unit 83, and the azimuth angle converted by the second conversion processing unit 85. The elevation angle, the azimuth angular velocity, and the elevation angular velocity are compared to determine whether the target observed by the observation sensor 3-m and the target detected by the target detector 32 are the same. The processing to be carried out is carried out.

  In FIG. 24, the image sensor 1-n, the observation sensor 3-m, the image generation unit 31, the target detection unit 32, the second tracking processing unit 34-m, the multi-sensor tracking processing unit 35, which are the components of the monitoring device, Display processing unit 40, first correspondence information storage unit 81, first conversion processing unit 82, first tracking processing unit 83, second correspondence information storage unit 84, second conversion processing unit 85, determination processing unit It is assumed that each of 86 and the display 16 is realized by dedicated hardware as shown in FIG. That is, the image sensor 1-n, the observation sensor 3-m, the image generation circuit 41, the target detection circuit 42, the second tracking processing circuit 44, the multi-sensor tracking processing circuit 45, the display processing circuit 50, and the first correspondence information storage. It is assumed that the circuit 91, the first conversion processing circuit 92, the first tracking processing circuit 93, the second correspondence information storage circuit 94, the second conversion processing circuit 95, the determination processing circuit 96, and the display 16 are realized. is doing.

  Image generation circuit 41, target detection circuit 42, second tracking processing circuit 44, multi-sensor tracking processing circuit 45, display processing circuit 50, first conversion processing circuit 92, first tracking processing circuit 93, second conversion The processing circuit 95 and the determination processing circuit 96 are, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

  The components other than the image sensor 1-n, the observation sensor 3-m, and the display 16 of the monitoring device are not limited to those realized by dedicated hardware, but software, firmware, or a combination of software and firmware. May be realized by.

  When the components other than the image sensor 1-n, the observation sensor 3-m, and the display 16 of the monitoring device are realized by software or firmware, the first correspondence information storage unit 81 and the second correspondence information storage unit 84 are stored. The video generation unit 31, the target detection unit 32, the second tracking processing unit 34-m, the multi-sensor tracking processing unit 35, the display processing unit 40, and the first conversion, which are configured on the memory 21 of the computer illustrated in FIG. A program for causing a computer to execute the processing procedure of the processing unit 82, the first tracking processing unit 83, the second conversion processing unit 85, and the determination processing unit 86 is stored in the memory 21, and the processor 22 of the computer stores the program in the memory 21. It suffices to execute the stored program.

Next, the operation will be described.
In the fifth embodiment, for simplification of description, it is assumed that the target detection unit 32 detects one target and the observation sensor 3-m observes one target.
FIG. 26 is an explanatory diagram showing reference positions of the image sensors 1-1 to 1-N and angles of the image sensors 1-1 to 1-N.
The first correspondence information storage unit 81 shows the correspondence relationship between the azimuth angle and the elevation angle whose origin is the reference position of the image sensors 1-1 to 1-N and the target object map coordinate position output from the target detection unit 32. Is stored.

The first conversion processing unit 82 converts the target object map coordinate position output from the target detection unit 32 into an azimuth angle and an elevation angle whose origin is the reference position of the image sensors 1-1 to 1-N.
That is, the first conversion processing unit 82 refers to the correspondence relationship stored in the first correspondence information storage unit 81, and the azimuth angle and the azimuth angle corresponding to the target object map coordinate position output from the target detection unit 32. The elevation angle is acquired, and the acquired azimuth angle and elevation angle are output to the first tracking processing unit 83.
The first tracking processing unit 83 performs the tracking processing of the azimuth angle and the elevation angle output from the first conversion processing unit 82, and the azimuth angle and the elevation angle after the tracking processing and the angular velocity of the azimuth angle after the tracking processing. And the angular velocity of the elevation angle are output to the determination processing unit 86.

Each of the second tracking processing unit 34-m and the multi-sensor tracking processing unit 35 performs the second conversion on the target object observation position after the tracking process and the target object observation speed that is the speed of the target object after the tracking process. Output to the processing unit 85.
The second conversion processing unit 85 uses the target object observation position output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35 as the reference position of the image sensors 1-1 to 1-N as the origin. Convert to azimuth and elevation.
That is, the second conversion processing unit 85 refers to the correspondence relationship stored in the second correspondence information storage unit 84, and is output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. The azimuth angle and the elevation angle corresponding to the target object observation position are acquired.

In addition, the second conversion processing unit 85 sets the target observation speed output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35 to the origin of the reference positions of the image sensors 1-1 to 1-N. To the angular velocity of azimuth and the angular velocity of elevation.
That is, the second conversion processing unit 85 refers to the correspondence relationship stored in the second correspondence information storage unit 84, and is output from the second tracking processing unit 34-m or the multi-sensor tracking processing unit 35. The angular velocity of azimuth and the angular velocity of elevation corresponding to the target object observation velocity are acquired.
The second conversion processing unit 85 outputs the acquired azimuth angle, elevation angle, azimuth angular velocity, and elevation angular velocity to the determination processing unit 86.

The determination processing unit 86 outputs the azimuth angle, the elevation angle, the angular velocity of the azimuth angle, and the angular velocity of the elevation angle after the tracking processing, which are output from the first tracking processing unit 83, and the azimuth angle output from the second conversion processing unit 85. The elevation angle, the azimuth angular velocity, and the elevation angular velocity are compared to determine whether the target observed by the observation sensor 3-m and the target detected by the target detector 32 are the same. To do.
Hereinafter, the processing content of the determination processing unit 86 will be specifically described.

式（１０）において、
ｘハットｘｔ，ＨＥＴは、第２の変換処理部８５から出力された目標物の時刻ｔにおける方位角、仰角、方位角の角速度及び仰角の角速度を含む推定状態ベクトル
ｘハットｘｔ，ＣＡＭは、第１の追尾処理部８３から出力された目標物の時刻ｔにおける方位角、仰角、方位角の角速度及び仰角の角速度を含む推定状態ベクトル
Ｐｔ，ＨＥＴは、第２の追尾処理部３４−ｍ又はマルチセンサ追尾処理部３５の追尾処理による目標物の時刻ｔにおける推定誤差を示す推定誤差共分散行列
Ｐｔ，ＣＡＭは、目標検出部３２により検出された目標物の時刻ｔにおける推定誤差を示す推定誤差共分散行列The determination processing unit 86 calculates the azimuth angle, the elevation angle, the angular velocity of the azimuth angle and the angular velocity of the elevation angle after the tracking processing, which is output from the first tracking processing unit 83, as shown in the following expression (10). Using the azimuth angle, the elevation angle, the angular velocity of the azimuth angle, and the angular velocity of the elevation angle output from the conversion processing unit 85, the test value ε of the chi-square test is calculated.

In equation (10),
x hat xt, HET is an estimated state vector including the azimuth angle, the elevation angle, the angular velocity of the azimuth angle, and the angular velocity of the elevation angle at the time t of the target output from the second conversion processing unit 85. x hat xt, CAM is the The estimated state vector Pt, HET including the azimuth angle, the elevation angle, the angular velocity of the azimuth angle, and the angular velocity of the elevation angle at the time t of the target output from the first tracking processing unit 83 is the second tracking processing unit 34-m or The estimation error covariance matrix Pt, CAM indicating the estimation error at the time t of the target object by the tracking processing of the sensor tracking processing unit 35 is the estimation error covariance matrix indicating the estimation error at the time t of the target object detected by the target detection unit 32. Covariance matrix

判定処理部８６は、カイ二乗検定の検定値εが閾値εｔｈ未満であれば、観測センサ３−ｍにより観測された目標物と、目標検出部３２により検出された目標物とが同一物であると判定する。
判定処理部８６は、カイ二乗検定の検定値εが閾値εｔｈ以上であれば、観測センサ３−ｍにより観測された目標物と、目標検出部３２により検出された目標物とが同一物でないと判定する。Next, the determination processing unit 86 compares the test value ε of the chi-square test with the preset threshold εth as shown in the following equation (11).

If the test value ε of the chi-square test is less than the threshold εth, the determination processing unit 86 determines that the target object observed by the observation sensor 3-m is the same as the target object detected by the target detection unit 32. To determine.
If the test value ε of the chi-square test is greater than or equal to the threshold εth, the determination processing unit 86 determines that the target object observed by the observation sensor 3-m and the target object detected by the target detection unit 32 are not the same. judge.

  As is clear from the above, the azimuth angle, the elevation angle, the angular velocity of the azimuth angle, and the angular velocity of the elevation angle after the tracking processing output from the first tracking processing unit 83 and the azimuth angle converted by the second conversion processing unit 85. , The elevation angle, the azimuth angular velocity, and the elevation angular velocity are compared to determine whether the target observed by the observation sensor 3-m is the same as the target detected by the target detector 32. The determination processing unit 86 for determining is provided. Thereby, even if the target is an aircraft in flight, it is possible to determine whether the target observed by the observation sensor 3 and the target in the captured image of the image sensor 1 are the same. There is an effect that can be done.

Sixth Embodiment
In the above-described fifth embodiment, an example of the monitoring device in which the first tracking processing unit 83 is provided after the first conversion processing unit 82 is shown.
However, this is only an example, and as shown in FIG. 27, it may be a monitoring device in which the first tracking processing unit 33 is provided in the preceding stage of the first conversion processing unit 82.
27 is a configuration diagram showing a monitoring device according to a sixth embodiment of the present invention.
Also in the case of the sixth embodiment, as in the case of the fifth embodiment, even when the target is an aircraft in flight, the target observed by the observation sensor 3 and the target in the captured image of the image sensor 1 It can be determined whether or not the object is the same object.

  In the invention of the present application, it is possible to freely combine the respective embodiments, modify any of the constituent elements of each of the embodiments, or omit any of the constituent elements of each of the embodiments within the scope of the invention. .

  The present invention relates to a monitoring device that determines whether a target observed by an observation sensor is the same as a target in an image captured by an image sensor.

  1, 1-1 to 1-N image sensor, 2 target detection unit, 3, 3-1 to 3-M observation sensor, 4 object shape information storage unit, 5 conversion processing unit, 6 first determination processing unit, 7 Display processing unit, 8 inclusion area specifying unit, 9 average position output unit, 11 target detection circuit, 12 shape memory circuit, 13 conversion processing circuit, 14 first determination processing circuit, 15 display processing circuit, 16 display, 21 memory, 22 processor, 31 image generation unit, 32, 32b target detection unit, 33 first tracking processing unit, 34-1 to 34-M second tracking processing unit, 35 multi-sensor tracking processing unit, 36 conversion processing unit, 37 First determination processing unit, 38 Determination result storage unit, 39 Smoothing processing unit, 40 Display processing unit, 41 Video generation circuit, 42 Target detection circuit, 43 First tracking processing circuit, 44 Second tracking processing circuit, 4 Multi-sensor tracking processing circuit, 46 conversion processing circuit, 47 first determination processing circuit, 48 determination result storage circuit, 49 smoothing processing circuit, 50 display processing circuit, 61 observation position projection unit, 62 second determination processing unit, 63 Third tracking processing unit, 64 Third determination processing unit, 65 Target selection unit, 71 Observation position projection circuit, 72 Second determination processing circuit, 73 Third tracking processing circuit, 74 Third determination processing circuit , 75 target selection circuit, 81 first correspondence information storage unit, 82 first conversion processing unit, 83 first tracking processing unit, 84 second correspondence information storage unit, 85 second conversion processing unit, 86 Judgment processing unit, 91 First correspondence information storage circuit, 92 First conversion processing circuit, 93 First tracking processing circuit, 94 Second correspondence information storage circuit, 95 Second conversion processing circuit, 96 Judgment processing circuit .

Claims (14)

An image sensor that captures an image of an area in which a target object is present and outputs a captured image of the area,
A target detection unit that detects a target object in the captured image output from the image sensor, and specifies a target object inclusion image area that is a range that includes the detected target object,
An observation sensor that observes a target object, and outputs the identification information of the observed target object and the target object observation position that is the position of the observed target object,
The size of the target object is specified by the identification information output from the observation sensor, and each of the specified size of the target object and the target object observation position output from the observation sensor is converted into a projection plane of the image sensor. Then, from the converted target size and the converted target observation position, a target inclusion that is a range that includes the target observed by the observation sensor on the projection surface of the image sensor. A conversion processing unit that specifies the observation area,
The target inclusion observation region identified by the conversion processing unit and the target inclusion image region identified by the target detection unit are compared, and the target observed by the observation sensor and the target detection unit are compared. A first determination processing unit that determines whether or not the detected target object is the same object.   When it is determined by the first determination processing unit that they are the same object, on the captured image output from the image sensor, from the target object observation position converted to the projection surface by the conversion processing unit and the observation sensor. The monitoring device according to claim 1, further comprising a display processing unit that displays each of the output identification information.   The monitoring device according to claim 2, wherein the display processing unit displays the target object inclusion image region specified by the target detection unit on the captured image output from the image sensor. On the projection plane of the image sensor, both inclusion areas that are both the target inclusion observation area specified by the conversion processing unit and the target inclusion image area specified by the target detection unit are specified. An inclusion area specifying unit that
The monitoring device according to claim 2, wherein the display processing unit displays both inclusion areas specified by the inclusion area specifying unit on the captured image output from the image sensor. On the projection surface of the image sensor, the target observation position converted into the projection surface by the conversion processing unit and the position of the target object in the captured image detected by the target detection unit are weighted averaged and weighted. Equipped with an average position output unit that outputs the averaged position,
The monitoring device according to claim 2, wherein the display processing unit displays the position output from the average position output unit on the captured image output from the image sensor. The observation sensor outputs observation error information indicating an observation error range of the target object observation position,
The monitoring device according to claim 1, wherein the conversion processing unit corrects the specified target inclusion observation region according to the observation error information output from the observation sensor. A smoothing processing unit for smoothing the target object inclusion image region specified by the target detecting unit in the time direction,
The monitoring device according to claim 2, wherein the display processing unit displays the target object inclusion image region smoothed by the smoothing processing unit on the captured image output from the image sensor. A first object for performing tracking processing of the target object detected by the target detection section, and outputting to the first determination processing section a target object inclusion image area that is a range including the target object after the tracking processing. Tracking processing unit of
A second tracking processing unit that performs tracking processing of the target object observed by the observation sensor and outputs a target object observation position that is the position of the target object after the tracking processing to the conversion processing unit;
The conversion processing unit converts each of the size of the specified target and the target observation position output from the second tracking processing unit into a projection plane of the image sensor, and the size of the converted target. From the converted target object observation position, on the projection surface of the image sensor, specify a target object inclusion observation region that is a range including the target object observed by the observation sensor,
The first determination processing unit compares the target object inclusion observation area specified by the conversion processing unit with the target object inclusion image area output from the first tracking processing unit, and the observation sensor detects The monitoring apparatus according to claim 1, wherein it is determined whether the observed target object and the target object detected by the target detection unit are the same object. The second tracking processing unit outputs estimation error information indicating an estimation error due to the tracking processing of the target object,
9. The monitoring device according to claim 8, wherein the conversion processing unit corrects the specified target inclusion observation region according to the estimation error information output from the second tracking processing unit. The first determination processing unit,
On the projection surface of the image sensor, the overlap range between the target inclusion observation region specified by the conversion processing unit and the target inclusion image region specified by the target detection unit is calculated, and the overlap range is a threshold value. If it is above, it is determined that the target object observed by the observation sensor and the target object detected by the target detection unit are the same object, and if the overlapping range is less than the threshold value, the observation sensor The monitoring device according to claim 1, wherein it is determined that the target observed by the target and the target detected by the target detecting unit are not the same. The first determination processing unit,
If a plurality of targets are detected by the target detection unit, and the target inclusion image area of each target is specified by the target detection unit, on the projection surface of the image sensor, the conversion processing unit. By calculating the overlapping range of the target inclusion observation area specified by, and the target inclusion image area of each of the targets, respectively, if the total of each overlapping range is equal to or more than a threshold value, the plurality of target objects are If it is a part of one target object, it is determined that the target object observed by the observation sensor and the one target object are the same, and the total of the overlapping ranges is less than the threshold value, The monitoring device according to claim 1, wherein it is determined that the target observed by the observation sensor and the one target are not the same. An observation position projection unit that calculates a target observation position output from the observation sensor, when moving in the line-of-sight direction of the image sensor, calculates a position intersecting the ground plane as a projection position,
By comparing the projection position calculated by the observation position projection unit and the target object map coordinate position which is the coordinate position on the map where the target object detected by the target detection unit is present, the observation sensor The monitoring according to claim 1, further comprising a second determination processing unit that determines whether the observed target object and the target object detected by the target detection unit are the same. apparatus. A second tracking processing unit that performs a tracking process of the target observed by the observation sensor and outputs a target observation position that is the position of the target after the tracking process;
Among the one or more target objects detected by the target detection unit, a tracking process of a target object other than the target objects determined to be the same by the second determination processing unit is performed, and after the tracking processing. A third tracking processing unit that outputs a target map coordinate position that is the position of the target of
The target observed by the observation sensor using the target observation position output from the second tracking processing unit and the target map coordinate position output from the third tracking processing unit, and A third determination processing unit that determines whether or not the target object detected by the target detection unit is the same object;
13. A target object selection unit that selects one or more target objects from among the target objects that are determined to be the same by the first to third determination processing units. The monitoring device described. The target selection unit,
Select one of the targets from the targets determined to be the same by the second determination processing unit,
A target object different from the selected target object is selected from among the target objects determined to be the same by the third determination processing unit,
14. The monitoring device according to claim 13, wherein a target different from the selected target is selected from the targets determined to be the same by the first determination processing unit.

Images