JP7669239B2 - Target detection device, target detection method, and program - Google Patents

Description

Embodiments of the present invention relate to a target detection device, a target detection method, and a program.

Guided flying objects are known that fly towards a target after being launched from the ground. Some of these types of flying objects are equipped with a target detection device that can detect an approaching target while the flying object itself is in flight. The target detection device processes image data acquired by an imaging device such as an infrared camera, and detects the target by extracting features such as brightness value and shape.

Patent No. 5713939

When a target approaches from a low altitude, a situation may occur in which the target is overlooked from a higher altitude than the target. If an attempt is made to capture an image of the target from a flying object in this situation, the complex background caused by the ground surface will take up most of the captured image. In such cases, the background captured will have a higher brightness value than the target, or a background similar in size and shape to the target, increasing the possibility of erroneously detecting the background as the target.
SUMMARY OF THE DISCLOSURE An object of the present invention is to provide a target detection device, a target detection method, and a program that are capable of reliably detecting a target regardless of the background.

The target detection device of the embodiment is a target detection device that can be mounted on a flying object. This target detection device includes an information acquisition unit, an imaging range calculation unit, an imaging device, an estimated image generation unit, a feature map generation unit, and a determination unit. The information acquisition unit acquires position information and attitude angle information of the flying object during flight. The imaging range calculation unit calculates the imaging range based on the position information and attitude angle information. The imaging device captures the imaging range and sequentially generates frame images. The estimated image generation unit predicts the next frame image based on the amount of displacement between the imaging range and the frame image to generate a predicted frame image. The feature map generation unit calculates the difference between the latest frame image and the predicted frame image, and cancels the background component from the latest frame image. The determination unit determines the target from the frame image from which the background component has been canceled.

FIG. 1 is a functional block diagram showing an example of a target detection system according to an embodiment. FIG. 2 is a functional block diagram illustrating an example of the processor 32 shown in FIG. FIG. 3 is a diagram showing an example of the area calculated by the imaging range calculation unit 32b. FIG. 4 is a diagram for explaining generation of a predicted frame image. FIG. 5 is a diagram for explaining the process of the feature map generating unit 32d. FIG. 6 is a flowchart showing an example of a processing procedure of the target detection device 30.

Hereinafter, embodiments will be described with reference to the accompanying drawings.
Fig. 1 is a functional block diagram showing an example of a target detection system according to an embodiment. The target detection system 1 shown in Fig. 1 can be mounted on, for example, a guided flying object.
The target detection system 1 includes an imaging device 10, an inertial device 20, and a target detection device 30. The target detection system 1 detects targets by applying image processing to image data acquired by the imaging device 10. In the image processing, the imaging range of the imaging device 10 is calculated from position information and attitude angle information of the target detection system 1 acquired by the inertial device 20, and the position coordinates of this imaging range on the earth's surface are used.

The imaging device 10 outputs captured image data to the target detection device 30. The imaging device 10 is, for example, a camera that captures images of a scene including a target in a time-series manner. The imaging device 10 may be, for example, an infrared camera equipped with an imaging element sensitive to the infrared range. Alternatively, the imaging device 10 may be a camera equipped with an imaging element sensitive to the visible light range.

The inertial device 20 is a sensor that acquires the position and attitude angle of the flying object on which the target detection device 30 is mounted. The inertial device 20 may be, for example, an inertial sensor that calculates the position and rotation from an accelerometer and a gyroscope. The inertial device 20 may be, for example, a GPS (Global Positioning System) that calculates the position using positioning signals from satellites.

The target detection device 30 includes an input device 31, a processor 32, a memory 33, a storage device 34, and an output device 35. In other words, the target detection device 30 is a computer.

The input device 31 is an interface device for inputting image data sequentially acquired by the imaging device 10 to the processor 32. The input device 31 is an interface device for inputting the position and attitude angles of the target detection system 1 sequentially acquired by the inertial device 20 to the processor 32.

The processor 32 executes the target detection program 34a stored in the storage device to detect a target in the captured image from the captured image, position, and attitude angle information acquired via the input device 31. The processor 32 is, for example, a CPU (Central Processing Unit). The processor 32 may be an MPU, a GPU, an ASIC, an FPGA, or the like. The processor 32 may be a single CPU, or multiple CPUs, or the like.

The memory 33 includes a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores the startup program for the target detection device 30 and various parameters necessary for the operation of the target detection device 30. The RAM is used as a working memory for various processes performed by the processor 32. For example, the RAM is used to store multiple captured images acquired via the input device 31.

The storage device 34 is a storage device that stores the target detection program 34a. The storage device 34 may be a flash memory or the like.

The output device 35 is an interface device for outputting information about targets detected by the processor 32 to the outside.

FIG. 2 is a functional block diagram showing an example of the processor 32 of FIG. 1. The processor 32 includes, as its processing functions, an imaging control unit 32a, an imaging range calculation unit 32b, an estimated image generation unit 32c, a feature map generation unit 32d, a target determination unit 32e, and an output control unit 32f. For example, the processor 32 executes a target detection program 34a to function as the imaging control unit 32a, the imaging range calculation unit 32b, the estimated image generation unit 32c, the feature map generation unit 32d, the target determination unit 32e, and the output control unit 32f.

The imaging control unit 32a controls the operation of the imaging device 10. The imaging control unit 32a controls, for example, the directivity angle of the imaging direction for the imaging device 10.

The imaging range calculation unit 32b sequentially calculates the coordinates of the ground surface in the reference coordinate system captured by the imaging device 10 for each pixel. The imaging range calculation unit 32b calculates the amount of displacement of the imaging range for each pixel between time series.

Figure 3 is a diagram showing an example of an area calculated by the imaging range calculation unit 32b. In Figure 3, the reference coordinate system 501 is an inertial coordinate system, and its origin is set to, for example, the launch point of the flying object, or the position 502a of the flying object in a previous frame. The line of sight direction and imaging range at the time of the previous frame are indicated by dotted lines, and the line of sight direction and imaging range at the time of the latest frame are indicated by solid lines.

Based on the directional angle of the imaging device 10 instructed by the imaging control unit 32a to the imaging device 10 and the position and attitude angle of the target detection system 1 acquired by the inertial device 20, the imaging range calculation unit 32b calculates the line of sight direction 503a of the previous frame and the line of sight direction 503b of the latest frame in the reference coordinate system 501.

The imaging range calculation unit 32b calculates the previous frame imaging range 504a and the latest frame imaging range 504b in the reference coordinate system 501 based on the calculated line of sight direction and the position information of the target detection system 1 acquired by the inertial device 20. The imaging displacement amount 505 is calculated as the difference for each pixel between the previous frame imaging range 504a and the latest frame imaging range 504b.

Returning to FIG. 2 again, the explanation will be continued.
The estimated image generating unit 32c generates an image (predicted frame image) that is to be captured in the latest frame by converting the image captured in the past frame based on the imaging displacement amount 505. That is, the estimated image generating unit 32c generates a predicted frame image by predicting the next frame image based on the calculated imaging range and the displacement amount between the past frame image.

Fig. 4 is a diagram for explaining generation of a predicted frame image. Fig. 4 shows a method for generating an image to be captured in the latest frame when the amount of capture displacement 505 is a translation of the captured image in the vertical and horizontal directions.
4, the latest frame imaging range 602 is an area shifted rightward and downward by the amount of imaging displacement 505 with respect to the past frame image 601. Based on this, the estimated image generating unit 32c translates (shifts) each of the pixel coordinates of the past frame image 601 by the amount of imaging displacement 505 to generate a predicted frame image 605 of the latest frame. At this time, an invalid area 604a where no destination exists in the past frame image 601 and an invalid area 604b where no source exists in the predicted frame image 605 are generated. Because no images exist in these invalid areas 604a and 604b, the effective field of view is narrowed by the amount of imaging displacement 505.

Figure 4 shows an example of a case where the imaging displacement amount 505 of all pixels translates the same amount in the vertical and horizontal directions of the image. On the other hand, in a real system, it is possible that the displacement amount in the direction in which the imaging background expands, the rotational displacement amount, and the displacement amount for each pixel are different. In this case, homography transformation is applied.

That is, a homography matrix is calculated from the past frame image 601 and the latest frame imaging range 602, and a homography transformation is performed on the valid area 603a of the past frame image 601, thereby generating a predicted frame image 605.

Returning to FIG. 2 again, the explanation will be continued.
The feature map generating unit 32d cancels the background component from the captured image by calculating the difference between the captured image of the latest frame acquired from the imaging device 10 and the predicted frame image 605 generated by the estimated image generating unit 32c. This emphasizes the target component to be detected.

Figure 5 is a diagram for explaining the processing of the feature map generating unit 32d. In Figure 5, first, a difference map 702 between the latest frame image 701 and the predicted frame image 605 is calculated.

The amount of displacement of the predicted frame image 605 is calculated with the entire imaging range as the ground surface background. On the other hand, since the flying target is flying at an altitude higher than the ground surface, the motion parallax between the background components and the target components in the image captured by the imaging device 10 is different. Therefore, by assuming the ground surface in the image and taking the difference between the predicted frame image 605, which has been aligned with the background, and the latest frame image 701, the background components can be cancelled. Furthermore, the target components with different motion parallax remain without being cancelled.

At this time, the target components remaining in the difference map 702 contain positive and negative component pixels, and are therefore divided into a positive component map 703 and a negative component map 704. The target imaged in the positive component map 703 is the target component of the latest frame image 701, and so the position at which the target is imaged is the latest target position. On the other hand, the target imaged in the negative component map 704 is a position transformed by the amount of displacement of the background component from the previous frame image 601, and so does not indicate the latest target position.

Therefore, the estimated image generating unit 32c inversely transforms the negative component map 704 to generate a negative component map (inverse transformation). Specifically, a transformation inverse to the transformation used to generate the predicted frame image 605 from the past frame image 601 is performed on the negative component map 704, a process is performed to return the target imaging position to the position of the past frame image 601, and the absolute value is taken to generate the negative component map (inverse transformation). Finally, the background component is cancelled out by taking the sum of the positive component map 703 and the negative component map (inverse transformation) 705. As a result, a feature map 706 is generated in which the target pixel is emphasized.

That is, the feature map generating unit 32d divides the difference map 702 into a positive component map 703 in which pixels of positive components are extracted, and a negative component map 704 in which pixels of negative components are extracted and the absolute values of the pixel values are mapped. The feature map generating unit 32d also inversely transforms the negative component map 704 to return the target pixel coordinates. Furthermore, the feature map generating unit 32d calculates the sum of the negative component map (inverse transformation) 705 and the positive component map 703 to generate a feature map in which the intensity of the target component is emphasized.

Returning to FIG. 2 again, the explanation will be continued.
The target determination unit 32e obtains the feature map in the latest frame from the feature map generation unit 32d. Then, the target determination unit 32e determines the coordinates of the final target in the image from the feature map. For example, the pixel with the maximum image luminance value is set as the target. In addition, the feature map may be subjected to known image processing such as binarization processing and edge detection, and the target coordinates may be determined based on the results.

The output control unit 32f calculates the error angle from the pixel coordinates of the target determined by the target determination unit 32e. The output control unit 32f controls the output of the error angle information to the outside. For example, the output control unit 32f outputs the target error angle information to the control system (propulsion system, steering mechanism, etc.) of the guided missile via the output device 35.

6 is a flowchart showing an example of a processing procedure of the target detection device 30. The processing of FIG.
In step S<b>101 , the processor 32 acquires the imaging direction from the imaging device 10 , and acquires the position and attitude angle of the target detection system 1 from the inertial device 20 .

In step S102, the processor 32 calculates the imaging range from the acquired imaging direction, position, and attitude angle.
In step S<b>103 , the processor 32 acquires the captured image of the latest frame from the imaging device 10 , and acquires the captured image of the previous frame from the memory 33 .

In step S104, the processor 32 generates an image (predicted frame image 605) that is estimated to be captured in the latest frame from the calculated imaging range and the captured image of the acquired past frame.
In step S105, the processor 32 cancels the background from the generated predicted frame image 605 and the captured image of the latest acquired frame, and generates a feature map in which the target is emphasized.

In step S106, the processor 32 determines a target pixel coordinate from the generated feature map, for example, determining the pixel having the highest value as the target pixel coordinate.
In step S107, the processor 32 calculates an error angle from the determined target pixel coordinates, and outputs the error angle via the output device 35 to the outside.

As described above, according to the embodiment, it is possible to cancel the background from the captured image of the latest frame and generate a feature map that emphasizes the target. This makes it possible to accurately detect the target even if there is a complex background caused by the ground surface.

Therefore, according to the embodiment, it is possible to accurately detect a target even when a complex background caused by the ground surface is captured. In other words, it is possible to provide a target detection device, a target detection method, and a program that can reliably detect a target regardless of the background.

Note that the present invention is not limited to the above embodiment. For example, at least one of the imaging control unit 32a, imaging range calculation unit 32b, estimated image generation unit 32c, feature map generation unit 32d, target determination unit 32e, and output control unit 32f shown in FIG. 2 may be configured with dedicated hardware, etc.

Although an embodiment has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included within the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

1... target detection system, 10... imaging device, 20... inertial device, 30... target detection device, 31... input device, 32... processor, 32a... imaging control unit, 32b... imaging range calculation unit, 32c... estimated image generation unit, 32d... feature map generation unit, 32e... target determination unit, 32f... output control unit, 33... memory, 34... storage device, 34a... target detection program, 35... output device.

Claims (6)

A target detection device that can be mounted on an aircraft,
an information acquisition unit that acquires position information and attitude angle information of the flying object during flight;
an imaging range calculation unit that calculates an imaging range based on the position information and the attitude angle information;
an imaging device that captures the imaging range and sequentially generates frame images;
an estimated image generating unit that predicts a next frame image based on an amount of displacement between the imaging range and the frame image, and generates a predicted frame image;
a feature map generating unit that calculates a difference between a latest frame image and the predicted frame image and cancels background components from the latest frame image;
a determination unit for determining a target from the frame image in which the background component has been cancelled ,
The feature map generation unit
calculating a difference between the latest frame image and the predicted frame image to generate a difference map in which a background component is cancelled from the latest frame image and a target component is left;
Dividing the difference map into a positive component map in which pixels of positive components are extracted, and a negative component map in which pixels of negative components are extracted and absolute values of pixel values are mapped,
inversely transforming the negative component map to return the target pixel coordinates;
A target detection device that calculates the sum of the inversely transformed negative component map and the positive component map to generate a feature map in which the intensity of the target component is emphasized . The target detection device according to claim 1, wherein the imaging range calculation unit assumes that the frame image is the ground surface and calculates imaging position coordinates in a reference coordinate system for each pixel of the frame image based on the position information and attitude angle information. The target detection device according to claim 2, wherein the imaging range calculation unit calculates the imaging position coordinates in a time series and calculates the amount of displacement of the imaging pixel coordinates between any time series in the imaging position coordinates. The target detection device according to claim 2, wherein the estimated image generation unit performs image processing and conversion on an image frame captured in the past by the imaging device based on the amount of displacement of the imaging position coordinates to generate the predicted frame image. A method for detecting a target using a target detection device that can be mounted on an aircraft, comprising:
The target detection device acquires position information and attitude angle information of the flying object during flight;
a target detection device calculating an imaging range based on the position information and the attitude angle information;
the target detection device captures the imaging range with an imaging device to sequentially generate frame images;
a target detection device predicting a next frame image based on an amount of displacement between the imaging range and the frame image to generate a predicted frame image;
A target detection device calculates a difference between a latest frame image and the predicted frame image, and cancels a background component from the latest frame image;
A target detection device determines a target from the frame image in which the background component has been cancelled;
a target detection device calculating a difference between the latest frame image and the predicted frame image, and generating a difference map in which a background component is cancelled from the latest frame image and a target component remains;
A target detection device divides the difference map into a positive component map in which pixels of positive components are extracted, and a negative component map in which pixels of negative components are extracted and absolute values of pixel values are mapped;
a target detection device inversely transforming the negative component map to return pixel coordinates of the target;
The target detection method includes a target detection device calculating a sum of the inverse transformed negative component map and the positive component map to generate a feature map in which the intensity of the target component is emphasized. A program executable by a computer that can be mounted on a spacecraft,
An instruction to cause the computer to acquire position information and attitude angle information of the flying object during flight;
an instruction to cause the computer to calculate an imaging range based on the position information and the attitude angle information;
An instruction to the computer to cause an imaging device to capture the imaging range and sequentially generate frame images;
an instruction to cause the computer to predict a next frame image based on an amount of displacement between the imaging range and the frame image, and generate a predicted frame image;
instructions to the computer to calculate a difference between a latest frame image and the predicted frame image to cancel background components from the latest frame image;
instructions for causing the computer to determine a target from the frame image with the background component cancelled;
instructions to cause the computer to calculate a difference between the latest frame image and the predicted frame image to generate a difference map from the latest frame image in which background components are cancelled and target components remain;
An instruction to cause the computer to divide the difference map into a positive component map in which pixels of positive components are extracted, and a negative component map in which pixels of negative components are extracted and absolute values of pixel values are mapped;
instructions to cause the computer to inverse transform the negative component map to return target pixel coordinates;
and instructions for causing the computer to calculate a sum of the inverse transformed negative component map and the positive component map to generate a feature map in which the intensity of the target component is enhanced.