JP6593995B2 - Airport monitoring device - Google Patents

Description

  The present invention relates to an airport monitoring device that supports airport surface control work.

  In the conventional airport monitoring device, in the airport monitoring device that supports airport surface control work, particularly, image recognition processing and character recognition processing are used to reduce the load of control work and to automate alarms (for example, Patent Documents). 1).

  Specifically, by recognizing the position and identification number of all aircraft and vehicles moving on the airport surface in an overcrowded airport surface, and automatically issuing a collision warning for objects that may collide In order to reduce the air traffic controller's control duties and improve the safety of the airport, the following airport monitoring devices are disclosed. Using the image data of video cameras installed at several locations on the airport surface and character recognition by image processing, it recognizes the flight name, vehicle number, moving direction and position of all aircraft and vehicles moving on the airport surface. Display with correlation with surface detection radar. In addition, it has a collision warning transmission device that automatically transmits a collision warning to the airport surface, and an aircraft / vehicle collision warning reception device that issues a warning in response to the transmitted information.

Japanese Patent No. 2973302

  However, for target object coordinate estimation using one camera, it is necessary to accurately determine the ground and the ground contact surface of the target object. In the preceding example, the center of gravity and the lower end of the detection area are used, but there are problems that the detection area is missing or affected by shadows, and the accuracy is reduced at night.

  An object of the present invention is to provide an airport monitoring apparatus that solves the above problems and can accurately determine the position of a target object in the coordinate estimation of the target object using a monocular camera.

The airport monitoring device according to the present invention is
Imaging means for photographing an airport surface including a predetermined target object and outputting an image signal of the airport surface;
Target object detection means for detecting the target object from the image signal of the captured image;
Alignment means for aligning the detected target object on the airport surface using a predetermined edge model, which is contour data of the target object, and outputting an image signal of the alignment result;
Based on the image signal of the alignment result, the coordinates of the contact point of the target object with the airport plane on the image captured by the imaging unit are estimated, and the coordinates of the contact point are converted to the position coordinates on the map. A position estimating means for calculating a position coordinate of the target object on the map by performing coordinate conversion;
The target object detection means determines an overlap between the detection frame based on the background difference and the detection frame based on the tracking when integrating the detection result based on the background difference of the target object and the tracking result based on the tracking of the target object. When it is determined that they overlap, based on the area of the detection frame by the background difference, the area of the detection frame by the tracking, and the area of overlap of the detection frame by the background difference and the detection frame by the tracking, A predetermined overlap rate is calculated, and based on the overlap rate, an integration process for integrating the detection result by the background difference and the detection result by the tracking is performed .

  According to the airport monitoring apparatus according to the present invention, the position estimation accuracy of the target object using the camera is robust to the sunshine conditions, and the accuracy is improved. In addition, since the accuracy is improved, matching with information such as other radars is facilitated.

It is the schematic which shows operation | movement of the airport monitoring apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the airport monitoring apparatus of FIG. It is a flowchart which shows the target object detection process by the target object detection part 2 of FIG. It is the schematic for demonstrating the tracking process by the target object detection part 2 of FIG. It is the schematic which shows an example of the overlap determination process by the target object detection part 2 of FIG. It is a flowchart which shows the precise alignment process using the two-dimensional edge model by the precise alignment part 3 of FIG. It is the schematic which shows the precise alignment process using the two-dimensional edge model by the precise alignment part 3 of FIG. It is a block diagram which shows the structure of the airport monitoring apparatus which concerns on Embodiment 2 of this invention. It is a flowchart which shows the precise alignment process using the three-dimensional edge model by the precise alignment part 3A of the airport monitoring apparatus of FIG. It is a block diagram which shows the structure of the airport monitoring apparatus which concerns on Embodiment 3 of this invention. It is a flowchart which shows the point light source detection process by 2 A of point light source detection parts of FIG. It is a flowchart which shows the precise alignment process using the two-dimensional light model by the precise alignment part 3B of FIG. It is the schematic for demonstrating the precise alignment process by the precise alignment part 3 of FIG. It is a block diagram which shows the structure of the airport monitoring apparatus which concerns on Embodiment 4 of this invention. It is a flowchart which shows the precise alignment process using the three-dimensional light model by 3C of precise alignment parts of FIG.

  Embodiments according to the present invention will be described below with reference to the drawings. In addition, in each following embodiment, the same code | symbol is attached | subjected about the same component.

Embodiment 1 FIG.
1 is a schematic diagram showing the operation of an airport monitoring apparatus according to Embodiment 1 of the present invention. FIG. 2 is a block diagram showing the configuration of the airport monitoring apparatus of FIG. In FIG. 2, the airport monitoring apparatus according to the first embodiment includes a camera 1, a target object detection unit 2, a precise alignment unit 3 based on a two-dimensional edge model, a position estimation unit 4, and an airport surface 3D data memory 5. And a result display unit 6, an aircraft information memory 7, and a two-dimensional edge model memory 8.

As shown in FIG. 1, in the first embodiment, an airport surface including an aircraft that is a target object is photographed with a camera 1, and an aircraft is detected using an image signal of an image obtained from the camera 1. Then, an aircraft edge model prepared in advance is applied to the detected aircraft area. Finally, converted from coordinates aircraft ground on a road surface to the position coordinates on the map, and outputs a detection result of the estimated position of the aircraft. Hereinafter, the airport monitoring apparatus according to the first embodiment will be described in detail with reference to FIG.

  In FIG. 2, the camera 1 captures an airport surface including an aircraft as a target object, and transfers an image signal of the captured image to the target object detection unit 2, the precise alignment unit 3, and the result display unit 6. The target object detection unit b2 detects a region in the airport plane of the aircraft from the image of the transferred image signal, and transfers the detected aircraft region to the precision alignment unit 3 and the result display unit 6 using a two-dimensional edge model. To do. The result display unit 6 is a liquid crystal display, for example, and displays an image of an input image signal.

FIG. 3 is a flowchart showing target object detection processing by the target object detection unit 2 of FIG. In FIG. 3, first, in step ST1, using the obtained image and the reference image (background image), the blur amount of a stationary object in the image is estimated. The amount of blur is calculated using the following expression when the background image I ₀ (u coordinate, v coordinate, color channel (3)) and the current frame image I _t (u coordinate, v coordinate, color channel (3)) are used. The amount of blurring (u _b , v _b ) to be satisfied is calculated.

  Here, in blur correction, an image is selected so that the difference value when the entire image is shifted is minimized (u, v).

Next, in step ST2, the aircraft is detected in consideration of the shake amount (u _b , v _b ). For the detection, a method using a background difference, a method for extracting feature amounts from an aircraft in advance, and detecting using learning data obtained by machine learning may be used.

  FIG. 4 is a schematic diagram for explaining the tracking process by the target object detection unit 2 of FIG.

Next, in step ST3, using the template image T (u coordinate, v coordinate, color channel) of the object detection result before one process as shown in FIG. 4, the image I _t (u, v, color channel) of the current frame is used. To calculate an object position satisfying the following expression.

  At this time, N represents the search range of the u coordinate axis, M represents the search range of the v coordinate axis, and the above equation outputs the coordinates (u, v) at which the SAD (Sum of Absolute Difference) is minimized. It becomes the formula to do.

Next, in step ST4, the detection result output in step ST2 and the detection result detected in step ST3 are integrated. First, an overlap determination between two detection frames is performed. Upper left and lower right coordinates of the detection frame of the object detection result are detected by R _det (l _u : upper left u coordinate, l _v : upper left v coordinate, _ru : lower right u coordinate, r _v : lower right v coordinate) and tracking If the upper left and lower right marks of the frame are R _track (l _u : upper left u coordinate, l _v : upper left v coordinate, _ru : lower right u coordinate, r _v : lower right v coordinate) Do.

FIG. 5 is a schematic diagram showing an example of overlap determination processing by the target object detection unit 2 of FIG. As shown in FIG. 5, when it is determined that they overlap (O = 1), it is further determined whether or not they are integrated by further determining the overlap rate. As shown in the following equation, the overlap rate is determined by the area area _det of the object detection frame output by the process of step ST2, the area area _{track of the} detection frame by tracking output by the process of step ST3, and the object detection frame and tracking. It is calculated from the area area _overlap of the overlapping area of the detection frames.

  When this area is used and the following conditions are met, an integration process is performed.

Here, TH is an overlap rate. The integration process differs depending on whether the number of integration frames is one or two or more.
(A) When there is one integrated frame: The width of the detection frame of background tracking is compared with the width of the detection frame of tracking, and when the width of the tracking detection frame is large, the width of the tracking detection frame is adopted.
(B) When there are two or more integrated frames: An integrated frame including all the frames is calculated.

  In the process of step ST5, a template image used for tracking is extracted based on the detection frame obtained in the process of step ST4. Through the above processing, the target object is detected, and the result is transferred to the precision alignment unit 3.

  Next, the two-dimensional edge model precise alignment unit 3 uses the aircraft edge model stored in the two-dimensional edge model memory 8 to perform alignment within the aircraft region obtained by the target object detection unit 2. Do.

  FIG. 6 is a flowchart showing the precision alignment process using the two-dimensional edge model by the precision alignment unit 3 of FIG.

  First, in step ST11, an edge model is acquired from the two-dimensional edge model memory 8. The two-dimensional edge model is contour data on a two-dimensional image, and retains contour data when photographed from various angles. The acquired two-dimensional edge model is held in the two-dimensional edge model memory 8. On the other hand, aircraft information, which is information such as the aircraft model, is obtained from multi-latation or the like, which is a monitoring system for measuring the position of the aircraft or the like, and stored in the aircraft information memory 7 in advance. Next, the precise alignment unit 3 selects an edge model that matches the precise alignment unit 3. Further, the selection of the edge model from the installation position of the camera 1 may be limited.

  FIG. 7 is a schematic diagram showing a precision alignment process using a two-dimensional edge model by the precision alignment unit 3 of FIG.

  In step ST12 of FIG. 6, using the edge model acquired in the process of step ST11, as shown in FIG. 7, for example, a chamfer matching method is used to align the edge image. In FIG. 7, m1 is a two-dimensional edge model, m2 is an edge image, m3 is an edge model after alignment, the two-dimensional edge model m1 and the edge image m2 are aligned, and the edge model m3 after alignment is aligned. Get.

Next, in step ST13, the alignment in the process of step ST12 is evaluated. The alignment evaluation value (error) error is the sum of the luminance values e on the edge subjected to the alignment, and is calculated from the edge image I _e and the edge coordinate e (with E coordinates) of the edge model as follows: Calculate with the formula.

Here, δ (I _e (edge _m )) is a function that outputs 1 when there is an edge and outputs 0 when there is no edge.

  Next, in step ST14, when the alignment evaluation value (error) error is equal to or smaller than a predetermined threshold value, it is determined that the alignment has been achieved, and the process ends. If NO in step ST14, the processing from step ST11 is repeated.

  Through the above processing, precise alignment is performed, and the image signal of the alignment result image is transferred to the position estimation unit 4.

Next, the position estimation unit 4 uses the image of the alignment result of the precise alignment unit 3 based on the two-dimensional edge model and the airport surface 3D data stored in advance in the airport surface 3D data memory 5.
Calculate the coordinates on the map for the position of the aligned aircraft. The position estimation unit 4 estimates a ground contact point with the airport plane on the image regarding the aircraft from the image of the alignment result from the precise alignment unit 3 using the two-dimensional edge model. Next, the position coordinate on the map is calculated by converting the position coordinate of the aircraft on the airport plane on the image into the position coordinate on the airport plane 3D data. The calculation result is transferred to the result display unit 6 and displayed. The result display unit 6 draws an image of the position of the aircraft as the target object obtained by the target object detection unit 2 on the image captured by the camera 1 and the position of the aircraft on the map estimated by the position estimation unit 4 on the map. To do.

As described above, according to the present embodiment, the accuracy of position estimation of the target object using the camera 1 is improved, and matching with information such as other radars is facilitated.
Embodiment 2. FIG.
FIG. 8 is a block diagram showing a configuration of an airport monitoring apparatus according to Embodiment 2 of the present invention. The airport monitoring apparatus according to the second embodiment is different from the airport monitoring apparatus according to the first embodiment in FIG.
(1) In place of the precise alignment unit 3 based on the two-dimensional edge model, the precise alignment unit 3 based on the three-dimensional edge model is provided.
(2) Instead of the two-dimensional edge model memory 8, a three-dimensional edge model memory 8A is provided.
Hereinafter, the differences will be described in detail.

The airport monitoring apparatus according to the second embodiment captures an airport surface including an aircraft as a target object with the camera 1, and detects an area of the aircraft using an image signal of an image obtained from the camera 1.
Then, a two-dimensional edge model is created from the three-dimensional edge model for the detected aircraft region, and the aircraft edge model is applied. Finally, converted from coordinates aircraft ground on a road surface of airport surface to the position coordinates on the map, and outputs a detection result.

  In the second embodiment, as compared with the first embodiment, the precision alignment unit 3A based on the three-dimensional edge model particularly has a three-dimensional edge model memory 8A in the area of the aircraft obtained by the target object detection unit 2. Alignment is performed using a two-dimensional edge model generated from the three-dimensional edge model of the aircraft stored in.

  FIG. 9 is a flowchart showing a precise alignment process using a three-dimensional edge model by the precise alignment unit 3A of the airport monitoring apparatus of FIG.

In step ST31 of FIG. 9, a two-dimensional edge model is generated from the three-dimensional edge model and stored in the three-dimensional edge model memory 8A. Here, the three-dimensional edge model is contour data of an aircraft in a three-dimensional space. A two-dimensional edge model of the aircraft is generated from the contour data of the aircraft in the three-dimensional space. On the other hand, information such as the aircraft model is obtained from multi-rotation or the like, which is a monitoring system for measuring the position of the aircraft, etc., and stored in the aircraft information memory 7. Select. The two-dimensional edge model is generated by setting the installation position of the camera 1 as an initial value and generating a two-dimensional edge model when the camera 1 is at that position. Assuming that the three-dimensional coordinate e _3D (x, y, z) of the edge of the three-dimensional edge model is used, the coordinate e (u, v) of the two-dimensional edge can be calculated by the following equation.

  Here, A represents an internal parameter of the camera 1, R represents a rotation matrix, and T represents a translation vector. Thereby, a two-dimensional edge model is obtained.

  Next, in step ST12, using the edge model acquired in step ST11, alignment is performed on the edge image using, for example, a chamfer matching method as shown in FIG.

Next, in step ST13, the alignment evaluation in step ST12 is performed. The alignment evaluation value (error) e _i is the sum of the luminance values e on the aligned edges, and is calculated from the edge image I _e and the edge coordinates edge of the edge model (having E coordinates). Calculate with the following formula.

Here, δ (I _e (edge _m )) is a function that outputs 1 when there is an edge and outputs 0 when there is no edge.

Next, in step ST14, when the alignment evaluation value (error) e _i is equal to or smaller than a predetermined threshold value, it is determined that the alignment is achieved, and the process ends. In addition, when it is NO in step ST14, it repeats from the process of step ST31.

  Through the above processing, precise alignment is performed, and an image of the alignment result is transferred to the position estimation unit 4.

  As described above, according to the present embodiment, by having a three-dimensional edge model, the number of data can be reduced, and an arbitrary two-dimensional edge model can be generated. The object position estimation accuracy is improved. In addition, matching with information such as other radars is facilitated.

Embodiment 3 FIG.
FIG. 10 is a block diagram showing a configuration of an airport monitoring apparatus according to Embodiment 3 of the present invention. The airport monitoring apparatus according to Embodiment 3 differs from the airport monitoring apparatus according to Embodiment 1 of FIG.
(1) Instead of the target object detection unit 2, a point light source detection unit 2A is provided.
(2) In place of the precise alignment unit 3 based on the two-dimensional edge model, a precise alignment unit 3B based on the two-dimensional light model is provided.
(2) Instead of the two-dimensional edge model memory 8, a two-dimensional write model memory 8B is provided.
Hereinafter, the differences will be described in detail.

In the third embodiment, an airport surface including an aircraft that is a target object is photographed with the camera 1, and a point light source group of the aircraft is detected using an image obtained from the camera 1. Then, an aircraft light model prepared in advance is applied to the detected point light source group. Finally, converted from coordinates aircraft ground on a road surface to the position coordinates on the map, and outputs a detection result of the position coordinates of the aircraft.

  As described above, the third embodiment is different from the first embodiment in the point light source detection unit 2A, the two-dimensional light model precise alignment unit 3B, and the two-dimensional light model memory 8B. The point light source detection unit 2A detects the point light source group of the aircraft from the image signal of the image captured by the camera 1, and transfers the position information of the detected point light source group to the precise alignment unit 3B using the two-dimensional light model.

  FIG. 11 is a flowchart showing point light source detection processing by the point light source detection unit 2A of FIG.

  First, in step ST1 of FIG. 11, the blur amount is estimated as in the first embodiment. Next, in step ST32, an aircraft point light source is detected from the captured image. The point light source is detected by a method using background difference, and the position and color component of the point light source are output. With the above processing, the point light source detection unit 2A transfers the position and color component of the point light source to the precision positioning unit 3B based on the two-dimensional light model.

  Next, the precise alignment unit 3B based on the two-dimensional light model includes the position information (including color components) of the point light source group obtained by the point light source detection unit 2A and the aircraft stored in advance in the two-dimensional light model memory 8B. Alignment is performed using a two-dimensional light model.

  FIG. 12 is a flowchart showing the precision alignment process using the two-dimensional light model by the precision alignment unit 3B of FIG.

  In step ST31 of FIG. 12, first, a two-dimensional light model is acquired from the two-dimensional light model memory 8B. The two-dimensional light model is data on the light installation position of the aircraft on the two-dimensional image, and the light installation position when captured from various angles is held in the two-dimensional light model memory 8B. On the other hand, information such as the aircraft model is obtained from multi-rotation or the like, which is a monitoring system for measuring the position of the aircraft or the like, and stored in the aircraft information memory 7 in advance. The precision positioning unit 3b selects a two-dimensional light model that matches the precise positioning unit 3b. Here, the selection of the two-dimensional light model from the installation position of the camera 1 may be limited.

  Next, in step ST32, the input image is aligned using the two-dimensional light model acquired in step ST31 as shown in FIG. This alignment is performed by obtaining a correlation between the position information of the point light source group obtained by the point light source detection unit 2A and the position information of the two-dimensional light model.

  Next, in step ST33, the alignment in the process of step ST32 is evaluated. The alignment evaluation value (error) error is the sum of the distances d to the aligned point light sources, and is calculated from the matched E point light source groups p and the model matched point e by the following equation: To do.

Next, in step ST14, when the error error _i is equal to or smaller than a predetermined threshold value, it is determined that the alignment has been achieved, and the process ends. In addition, when it is NO in step ST14, it returns to step ST31 and repeats a process.

  FIG. 13 is a schematic diagram for explaining the precision alignment processing by the precision alignment unit 3 of FIG. In FIG. 13, m30 is a two-dimensional light model, m31 is a two-dimensional edge when there is an aircraft, m32 is a night image, and m33 is a light model after alignment. That is, by aligning the two-dimensional edge m31 of the aircraft in the two-dimensional light model m30 and the night image m32, the light model m33 after alignment can be obtained.

Through the above processing, precise alignment is performed, and the aircraft included in the two-dimensional light model and the ground contact position on the road surface of the airport surface are transferred to the position estimation unit 4. The result display unit 6 displays the aircraft region and the light model. Transfer and display the aircraft and the ground contact position on the road.

  As described above, according to the present embodiment, since the point light source detection unit 2A and the two-dimensional light model precise alignment unit 3B are provided, the target object position estimation accuracy using the camera 1 at night is improved. To do. In addition, matching with information such as other radars is facilitated.

Embodiment 4 FIG.
FIG. 14 is a block diagram showing a configuration of an airport monitoring apparatus according to Embodiment 4 of the present invention. The airport monitoring apparatus according to the fourth embodiment is different from the airport monitoring apparatus according to the third embodiment in FIG.
(1) Instead of the precise alignment unit 3B based on the two-dimensional light model, a precise alignment unit 3C based on the three-dimensional light model is provided.
(2) A three-dimensional light model memory 8C is provided instead of the two-dimensional light model memory 8B.
Hereinafter, the differences will be described in detail.

In the fourth embodiment, an airport surface including an aircraft that is a target object is photographed with the camera 1, and a point light source group of the aircraft is detected using an image obtained from the camera 1. Then, a two-dimensional edge model is created from the three-dimensional edge model for the detected point light source group of the aircraft, and the edge model of the aircraft is applied. Finally, converted from coordinates aircraft ground on a road surface to the position coordinates on the map, and outputs a detection result of the position of the aircraft.

  As described above, the fourth embodiment is different from the third embodiment in the precision positioning unit 3C using the three-dimensional light model and the three-dimensional light model memory 8C. The precise alignment unit 3C using the three-dimensional light model performs alignment using the aircraft point light source group obtained by the point light source detection unit 2A and the aircraft three-dimensional light model.

  FIG. 15 is a flowchart showing the precision alignment process using the three-dimensional light model by the precision alignment unit 3C of FIG.

  In step ST41 of FIG. 15, a two-dimensional light model is generated from the three-dimensional light model. The three-dimensional light model is the light coordinates of the aircraft in a three-dimensional space. A two-dimensional light model is generated from the light coordinates in the three-dimensional space. On the other hand, information such as the aircraft model is obtained from multi-rotation or the like, which is a monitoring system for measuring the position of the aircraft or the like, and stored in the aircraft information memory 7 in advance. The precision positioning unit 3C selects a three-dimensional light model that matches the precise positioning unit 3C. The two-dimensional light model is generated by the same method as in the second embodiment. Step ST32 and subsequent steps are executed in the same manner as in the third embodiment. Through the above processing, precise alignment is performed, and an alignment result including aircraft position information, which is a result of aircraft alignment, is transferred to the position estimation unit 4. The processing of the position estimation unit 4 is the same as that in the third embodiment.

  As described above, according to the present embodiment, by having the three-dimensional light model, the number of data can be reduced, and an arbitrary two-dimensional light model can be generated. The position estimation accuracy of the target object using is improved. In addition, matching with information such as other radars is facilitated.

DESCRIPTION OF SYMBOLS 1 Camera, 2 Target object detection part, 2A Point light source detection part, 3, 3A, 3B, 3C Precision alignment part, 4 Position estimation part, 5 Airport surface 3D data memory, 6 Result display part, 7 Aircraft information memory, 8 2D edge model memory, 8A 3D edge model memory, 8B 2D light model memory, 8C 3D light model memory, m1 2D edge model, m2 edge image, m3 Edge model after alignment, m30 2D light model , M31 Two-dimensional edge when there is an aircraft, m32 Night image, m33 Light model after alignment.

Claims (5)

Imaging means for photographing an airport surface including a predetermined target object and outputting an image signal of the airport surface;
Target object detection means for detecting the target object from the image signal of the airport surface ;
Using the predetermined edge model that is the contour data of the target object, the detected target object is aligned within the area of the target object detected by the target object detecting means, and an image signal of the alignment result is output. Alignment means to perform,
Based on the image signal of the alignment result, the coordinates of the contact point of the target object with the airport plane on the image captured by the imaging unit are estimated, and the coordinates of the contact point are converted to the position coordinates on the map. A position estimating means for calculating a position coordinate of the target object on the map by performing coordinate conversion ;
The target object detection means determines an overlap between the detection frame based on the background difference and the detection frame based on the tracking when integrating the detection result based on the background difference of the target object and the tracking result based on the tracking of the target object. When it is determined that they overlap, based on the area of the detection frame by the background difference, the area of the detection frame by the tracking, and the area of overlap of the detection frame by the background difference and the detection frame by the tracking, An airport monitoring apparatus that calculates a predetermined overlap rate and performs integration processing for integrating the detection result based on the background difference and the detection result based on the tracking based on the overlap rate . The target object detection means includes
(A) When the predetermined integrated frame used in the integration process is one, the detection frame based on the background difference is compared with the width of the detection frame based on the tracking. If the width of the detection frame based on the tracking is large, the tracking Adopt the width of the detection frame,
(B) The airport monitoring apparatus according to claim 1, wherein when the number of integrated frames is two or more, the integrated processing is performed by calculating an integrated frame including all integrated frames. Airport surveillance apparatus according to claim 1 or 2, characterized by further comprising display means for displaying the detection results and the position of the estimated target object of the target object obtained by the detection. The airport monitoring apparatus according to any one of claims 1 to 3, wherein the edge model is a two-dimensional edge model that is contour data of a target object on a two-dimensional image. The airport monitoring device according to any one of claims 1 to 3, wherein the edge model is a three-dimensional edge model that is contour data of a target object on a three-dimensional image.

Images