JP7566613B2 - Object detection device, object detection method, and object detection program - Google Patents

Description

Embodiments of the present invention relate to an object detection device, an object detection method, and an object detection program.

There is known technology for detecting objects moving through the air, for example, by processing infrared images captured by an imaging device such as an infrared camera.

Patent No. 5713939 Patent No. 6042146

Object Contour Extraction Based on Region Separability, IEICE Transactions on Computer Vision, Vol. J80-D2, No. 6, pp. 1406-1414

In a complex background that contains multiple objects such as the ground and clouds, the shape of the background area detected by image processing may change due to changes in contrast and the effects of noise. In such cases, this change in the background area may be mistakenly detected as the movement of the target object. It is also possible to detect the target object by specifying a luminance band, but even in this case, there is a possibility that a continuous area of high luminance values in the complex background may be mistakenly detected as the target object.

The embodiments provide an object detection device, an object detection method, and an object detection program that can accurately detect a target object even in a complex background.

An object detection device according to one embodiment includes a feature extraction unit, a tracking unit, and a determination unit. The feature extraction unit extracts candidate regions of an object in an image and likelihoods of the candidate regions based on feature amounts of the images acquired in time series from an imaging device. The tracking unit tracks candidate regions in a current image that are highly likely to be the object by accumulating the likelihoods extracted in time series. The determination unit determines a final position of the object based on the candidate regions tracked by the tracking unit. The tracking unit adds a likelihood value in a second candidate region extracted based on a current image to a frame likelihood map having a likelihood value in a first candidate region extracted based on a past image, subtracts a certain value from the likelihood value at coordinates of the frame likelihood map excluding the second candidate region, and tracks the object based on a comprehensive likelihood map calculated by the addition and subtraction.

FIG. 1 is a diagram showing an example of an object detection system including an object detection device according to each embodiment. FIG. 2 is a functional block diagram of the object detection device according to the first embodiment. FIG. 3A is a diagram showing an example of a search area. FIG. 3B is a diagram showing an example of a search region. FIG. 4A is a diagram showing an example of an image input in the feature extraction process. FIG. 4B is a diagram showing an example of a result of the feature extraction process. FIG. 5 is a diagram illustrating an example of feature amount tracking processing according to the first embodiment. FIG. 6 is a flowchart illustrating an example of processing performed by the object detection device according to the first embodiment. FIG. 7 is a flowchart showing the process of calculating the overall likelihood map in the first embodiment. FIG. 8 is a diagram showing an outline of control of the imaging direction. FIG. 9A is a diagram showing an overview of the control of the angle of view. FIG. 9B is a diagram showing an overview of the control of the angle of view. FIG. 10 is a diagram illustrating an example of feature amount tracking processing in the modified example of the first embodiment. FIG. 11 is a flowchart showing a process of calculating the overall likelihood map in the modified example of the first embodiment. FIG. 12 is a functional block diagram of an object detection device according to the second embodiment. FIG. 13 is a diagram illustrating an example of the search range setting process. FIG. 14A is a diagram showing a modified example of the search range setting process. FIG. 14B is a diagram showing a modified example of the search range setting process. FIG. 15 is a flowchart illustrating an example of processing performed by the object detection device according to the second embodiment.

(First embodiment)
The first embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of an object detection system including an object detection device according to each embodiment. The object detection system 1 includes an imaging device 10 and an object detection device 20. The object detection system 1 is provided in, for example, a control system of a flying object or the flying object itself. The object detection system 1 detects a target object by performing image processing on an image acquired using the imaging device 10. The object is, for example, a moving object such as a flying object. Here, in FIG. 1, configurations other than the object detection system 1 are omitted. For example, the flying object on which the object detection system 1 is mounted may have a propulsion device for obtaining a propulsive force, a steering device for changing a moving direction, and the like. The control system of the flying object may also be provided in the flying object itself.

The imaging device 10 is a camera that sequentially captures images of a scene including an object. For example, the imaging device 10 may be an infrared camera having an imaging element sensitive in the infrared region. The imaging device 10 may be a camera having an imaging element sensitive in the visible light region. The imaging device 10 generates an image having a luminance value according to the intensity of light received by each pixel of the imaging element. The imaging device 10 then outputs the generated image to the object detection device 20. Here, the imaging device 10 may have a pan/tilt mechanism so that the imaging direction can be changed. The imaging device 10 may also have a zoom optical system so that the angle of view can be changed. The imaging device 10 may be configured to be able to change only either the imaging direction or the angle of view.

The object detection device 20 has an input device 201, a processor 202, a memory 203, a storage device 204, and an output device 205.

The input device 201 is an interface device for inputting images sequentially acquired by the imaging device 10 to the processor 202.

The processor 202 detects an object in an image acquired via the input device 201 by executing an object detection program 2041 stored in the storage device 204. The processor 202 is, for example, a CPU. The processor 202 may be an MPU, a GPU, an ASIC, an FPGA, or the like. The processor 202 may be a single CPU, or a plurality of CPUs, or the like.

The memory 203 includes a ROM and a RAM. The ROM stores the startup program of the object detection device 20 and various parameters necessary for the operation of the object detection device 20. The RAM can be used as a working memory for various processes performed by the processor 202. For example, the RAM is used to temporarily store a likelihood map, which will be described later.

The storage device 204 is a storage device that stores the object detection program 2041. The storage device 204 may be a flash memory or the like.

The output device 205 is an interface device for outputting information about the object detected by the processor 202 to the outside as necessary. For example, the output device 205 includes a communication interface for transmitting information required for the control system of the flying object.

2 is a functional block diagram of the object detection device 20 according to the first embodiment. As shown in FIG. 2, the object detection device 20 of this embodiment includes an image capture control unit 301, a feature extraction unit 302, a tracking unit 303, a determination unit 304, and an output control unit 305. For example, the processor 202 operates as the image capture control unit 301, the feature extraction unit 302, the tracking unit 303, the determination unit 304, and the output control unit 305 by executing the object detection program 2041. The image capture control unit 301, the feature extraction unit 302, the tracking unit 303, the determination unit 304, and the output control unit 305 may be configured by dedicated hardware or the like.

The imaging control unit 301 controls the operation of the imaging device 10. The imaging control unit 301 instructs the imaging device 10 on the imaging direction. The imaging control unit 301 also instructs the imaging device 10 to change the angle of view, i.e., to zoom in or out. For example, the imaging control unit 301 pans or tilts the imaging device 10 so that the coordinates with high likelihood, which will be described later, become the center of the image captured in the next frame. For example, the imaging control unit 301 also zooms in or out the imaging device 10 depending on the level of likelihood. For example, the imaging control unit 301 zooms in the imaging device 10 when the likelihood is high, and zooms out the imaging device 10 when the likelihood is low. The imaging control unit 301 also acquires images input in time series from the imaging device 10 via the input device 201, and transmits the acquired images to the feature extraction unit 302.

The feature extraction unit 302 performs feature extraction processing. The feature extraction processing is processing for extracting candidate regions and likelihoods of objects from features in images acquired in time series by the imaging control unit 301. The likelihoods are values that represent the likelihood of an object.

For example, the feature extraction unit 302 sets a search area in the image. The search area has, for example, a central area 401 and a peripheral area 402 as shown in FIG. 3A. The central area 401 is a 2×2 pixel area. The peripheral area 402 is a one pixel area outside the central area 401. The size of the search area is not limited to the size shown in FIG. 3A. For example, as shown in FIG. 3B, the peripheral area 402 may be a two pixel area outside the central area 401. Multiple search areas may be set in the image. These multiple search areas may be the same size or different sizes.

After setting the search areas, the feature extraction unit 302 calculates the degree of separation as a feature for each search area. The degree of separation is the ratio of the variance of the brightness values of the central area 401 and the peripheral area 402. The feature extraction unit 302 then extracts candidate areas from the search areas based on the degree of separation. For example, the feature extraction unit 302 extracts search areas with a high degree of separation as candidate areas. There may be one or more candidate areas extracted.

Figures 4A and 4B are examples of feature extraction processing. Figure 4A is an example of an image input to the feature extraction unit 302. Image 500 shown in Figure 4A is an image that includes an object O. Figure 4B is an example of an image that has been subjected to feature extraction processing. For example, candidate regions 501, 502, and 503 are extracted in Figure 4B. Of candidate regions 501, 502, and 503, candidate region 501 is an area that is extracted as a candidate region because it includes object O in its central region. Candidate regions 502 and 503 are areas that are extracted as candidate regions due to the influence of the background.

After extracting candidate regions as shown in FIG. 4B, the feature extraction unit 302 sends the upper left X coordinates X1, X2, X3, the upper left Y coordinates Y1, Y2, Y3, the widths W1, W2, W3, the heights H1, H2, H3, and the likelihoods S1, S2, S3 of the candidate regions 501, 502, 503 to the tracking unit 303. Here, the likelihoods S1, S2, S3 may be, for example, the degree of separation.

Here, the feature extraction unit 302 is not limited to a configuration that extracts candidate regions based on the degree of separation. The feature extraction unit 302 may extract candidate regions using known image processing such as binarization of luminance values, Otsu's binarization (binarization based on the degree of separation calculated by the ratio of the inter-class variance and the intra-class variance between the central region and the peripheral region), and edge detection. The feature extraction unit 302 may also extract candidate regions by extracting differences from the image of the previous frame, such as inter-frame differences.

The feature extraction unit 302 may also remove from the candidate regions any of the extracted candidate regions in which either or both of the likelihood value and the average brightness value of the central region are lower than a preset value. A low likelihood value means that the central region and the peripheral regions are not well separated, i.e., the likelihood of the object being low. A low brightness value in the central region means that the candidate region is highly likely to be the background. Therefore, by removing from the candidate region any of the regions in which either or both of the likelihood value and the average brightness value of the central region are lower than a preset value, it is possible to prevent the background region from being erroneously extracted as the object region. The brightness value that is the threshold value when removing the candidate region may not be a predetermined value, but may be the difference between the average brightness values of the central region and the peripheral region. This allows the threshold to be automatically set for each candidate region.

The tracking unit 303 performs feature tracking processing. The feature tracking processing is processing for tracking an object based on an overall likelihood map calculated based on the top left X coordinate, top left Y coordinate, width, height, and likelihood as information on the candidate area sent from the feature extraction unit 302. The overall likelihood map is calculated from frame likelihood maps of multiple frames. The overall likelihood map and the frame likelihood map are maps in which each coordinate has a likelihood value.

Figure 5 is a diagram showing an example of feature tracking processing in the first embodiment. The tracking unit 303 calculates a frame likelihood map 600a in the current frame from an image 500a including the extraction result of the feature input in the current frame t. The frame likelihood map 600a in the frame t is calculated by adding the likelihood values in the candidate areas 501a, 502a, and 503a extracted in the image 500a to a likelihood map of initial values stored in advance. The likelihood map of initial values is a map in which the likelihood values in all coordinates are initial values, for example, zero. Specifically, the tracking unit 303 adds the likelihood values of each coordinate included in the candidate area 501a to each coordinate of the candidate area 601a corresponding to the candidate area 501a in the likelihood map of initial values. Similarly, the tracking unit 303 adds the likelihood value of each coordinate included in the candidate area 502a to the coordinates of the candidate area 602a corresponding to the candidate area 502a in the initial value likelihood map, and adds the likelihood value of each coordinate included in the candidate area 503a to the coordinates of the candidate area 603a corresponding to the candidate area 503a in the initial value likelihood map. Here, in FIG. 5, the difference in the likelihood value is shown by applying different hatching to each candidate area. In the example of FIG. 5, the likelihood value of the candidate area 601a is higher than the likelihood values of the candidate areas 602a and 603a.

After calculating the frame likelihood map 600a for frame t, the tracking unit 303 calculates the overall likelihood map 700a for the current frame t from the frame likelihood map 600a and the overall likelihood map 700b calculated for the previous frame t-1. Specifically, the tracking unit 303 adds the likelihood value of each coordinate in the frame likelihood map 600a to the likelihood value of the corresponding coordinate in the overall likelihood map 700b. Meanwhile, the tracking unit 303 subtracts a preset constant value from the added overall likelihood map 700b for coordinates that do not have a likelihood value in the frame likelihood map 600a, i.e., coordinates other than the candidate areas 601a, 602a, and 603a. In this way, the tracking unit 303 calculates the overall likelihood map 700a.

For example, in FIG. 5, the tracking unit 303 adds the likelihood values of the candidate regions 601a, 601b, and 601c in the frame likelihood map 600a to the likelihood values of the corresponding coordinates in the overall likelihood map 700b. Furthermore, the tracking unit 303 subtracts a preset constant value from the regions other than the candidate regions 601a, 602a, and 603a in the overall likelihood map 700b.

Here, the candidate areas 701b, 702b, 703b, 704b, 705b, and 706b having likelihood values in the overall likelihood map 700b do not match any of the candidate areas 601a, 602a, and 603a having likelihood values in the frame likelihood map 600a. Therefore, a fixed value is subtracted from the likelihood values of the coordinates corresponding to the candidate areas 701b, 702b, 703b, 704b, 705b, and 706b in the frame likelihood map 600a. In the overall likelihood map 700a in FIG. 5, in addition to the candidate areas 701a, 702a, and 703a corresponding to the candidate areas 601a, 602a, and 603a, only the candidate area 704a corresponding to the candidate area 703b, the candidate area 705a corresponding to the candidate area 704b, and the candidate area 706a corresponding to the candidate area 706b have likelihood values. In addition, the likelihood values of candidate areas 704a, 705a, and candidate area 706a are smaller than the original values.

In the overall likelihood map calculated in this way, if a candidate area with the same coordinates as a candidate area in the likelihood map one frame ago is extracted in the current frame, the likelihood of that candidate area will be higher. On the other hand, if a candidate area with different coordinates from the candidate area in the likelihood map one frame ago is extracted in the current frame, the likelihood of the candidate area extracted in the previous frame will be lower. For example, if the imaging device 10 is controlled so that a candidate area with a high likelihood of being the object is at the center of the image, this candidate area with a high likelihood hardly moves from the center of the image. In addition, it is considered that the object hardly changes in shape over time. On the other hand, the position of the candidate area for the background such as clouds may change due to changes in position and shape over time. For this reason, in the feature tracking process of the embodiment, the likelihood of a candidate area with a high probability of including the target object tends to increase over time, and even if a background similar to the object is extracted as a candidate area, the likelihood of the candidate area of this background tends to decrease over time. In this way, in the embodiment, even if a background area similar to the target object is erroneously extracted, the likelihood of the erroneously extracted background area can be reduced, and as a result, the target object can be detected with high accuracy.

The tracking unit 303 sends the coordinates of the candidate area having the highest likelihood in the overall likelihood map 700a calculated as described above and the likelihood value of that candidate area to the determination unit 304. In the example of FIG. 5, the tracking unit 303 sends the upper left X coordinate, upper left Y coordinate, width, height, and likelihood of the candidate area 701a to the determination unit 304.

Here, if no overall likelihood map is held for the previous frame, the frame likelihood map for the previous frame is used as the overall likelihood map. In this case, the likelihood values of the frame likelihood map are used as they are for the overall likelihood map. In the example of FIG. 5, there is no frame prior to image 500c in the t-2 frame. In this case, frame likelihood map 600c calculated based on image 500c is used as the overall likelihood map for the previous frame for calculating overall likelihood map 700b in the t-1 frame. Overall likelihood map 700b is calculated from frame likelihood map 600b calculated based on image 500b in the t-1 frame and frame likelihood map 600c.

In addition, if a frame likelihood map is not stored for the previous frame, i.e., if the current frame is the first frame, the aforementioned initial value likelihood map stored in advance may be used as the overall likelihood map.

In addition, in FIG. 5, the frame likelihood maps 600a, 600b, 600c and the overall likelihood maps 700a, 700b are shown as images in which the likelihood value is the brightness value. However, the frame likelihood map and the overall likelihood map do not necessarily need to be visualized. It is sufficient for the frame likelihood map and the overall likelihood map to have a likelihood value for each coordinate.

In addition, in the embodiment, the imaging device 10 is assumed to be controlled so that a candidate area with a high likelihood is located at the center of the image. However, if the movement of the object is small compared to the interval between imaging frames, the movement of the object in the image between frames is also small, and as a result, the likelihood value tends to increase over time. When tracking an object with small movement compared to the interval between such imaging frames, the imaging device 10 does not need to be controlled so that a candidate area with a high likelihood is located at the center of the image.

The determination unit 304 acquires the overall likelihood map for the current frame from the tracking unit 303. The determination unit 304 then determines the final position of the object in the image from the overall likelihood map. Specifically, the determination unit 304 determines the final position of the object depending on whether the highest likelihood value in the overall likelihood map is equal to or greater than a certain value. For example, if the detected likelihood value is equal to or greater than a certain value, the determination unit 304 determines the coordinates of the candidate area having that likelihood as the final position of the object. On the other hand, for example, if the highest likelihood value in the overall likelihood map is not equal to or greater than a certain value, the determination unit 304 does not determine the coordinates of the candidate area having that likelihood as the final position of the object.

The output control unit 305 controls the output of information about the position of the object determined by the determination unit 304. For example, the output control unit 305 outputs the information about the position of the object to the control system of the flying object via the output device 204.

Figure 6 is a flowchart showing an example of processing of the object detection device 20 in the first embodiment. The processing of Figure 6 is performed by, for example, the processor 202.

In step S101, the processor 202 acquires an image from the imaging device 10.

In step S102, the processor 202 extracts features of the acquired image and extracts candidate regions and their likelihoods based on the features.

In step S103, the processor 202 determines whether or not a feature has been extracted. If it is determined in step S103 that a feature has been extracted, the process proceeds to step S104. If it is determined in step S103 that there is no search area with a high degree of separation as a candidate area, for example, the process returns to step S101. In this case, the processor 202 acquires an image of the next frame and repeats the same process.

In step S104, the processor 202 calculates a frame likelihood map for the current frame from the coordinates and likelihood of each candidate region.

In step S105, the processor 202 calculates an overall likelihood map for the current frame from the frame likelihood map for the current frame. Here, the process of calculating the overall likelihood map is described.

Figure 7 is a flowchart showing the calculation process of the overall likelihood map in the first embodiment. In step S201, the processor 202 obtains a likelihood value from coordinates corresponding to a candidate area in the frame likelihood map in the current frame.

In step S202, the processor 202 adds the acquired likelihood value to the overall likelihood map calculated in the previous frame. As described above, if there is no overall likelihood map for the previous frame, the frame likelihood map for the previous frame may be used as the overall likelihood map. Furthermore, when the current frame is the first frame, the initial value of the likelihood map may be used as the overall likelihood map.

In step S203, the processor 202 subtracts a constant value from the likelihood value of each coordinate outside the candidate area of the frame likelihood map in the current frame in the overall likelihood map calculated in the previous frame. In this manner, the overall likelihood map in the current frame is calculated.

In step S204, the processor 202 extracts the coordinates and likelihood information of the candidate region with the highest likelihood from the overall likelihood map in the current frame. The processor 202 then ends the processing of FIG. 7.

Now, let us return to the explanation of FIG. 6. In step S106 after calculating the overall likelihood map for the current frame, the processor 202 determines whether the extracted likelihood is equal to or greater than a certain value. If it is determined in step S106 that the likelihood is not equal to or greater than the certain value, the process proceeds to step S107. If it is determined in step S106 that the likelihood is equal to or greater than the certain value, the process proceeds to step S109.

In step S107, the processor 202 controls the imaging direction of the imaging device 10 so that the candidate area with the highest likelihood is the center of the image captured in the next frame. That is, the processor 202 pans or tilts the imaging device 10. FIG. 8 is a diagram showing an overview of the control of the imaging direction. The processor 202 acquires, for example, the coordinates of the candidate area 701a with the highest likelihood in the overall likelihood map 700a generated from the current frame 500a. Then, the processor 202 calculates the distance between the images, for example, between the coordinates of the center of the acquired candidate area 701a and the coordinates of the center of the image. Then, the processor 202 determines the pan amount and tilt amount of the imaging device 10 based on the distance between the images, and pans and tilts the imaging device 10 according to the determined pan amount and tilt amount. In this way, in the image 500e captured in the next frame, the candidate area 501e corresponding to the candidate area 701a with the highest likelihood in the previous frame becomes the image center. The candidate area with the highest likelihood is an area with a high probability that the object O exists. By capturing an image so that the area where there is a high probability that the object O is present is at the center of the image, the possibility of the object O being lost from the captured image is reduced.

In step S108, the processor 202 controls the angle of view of the imaging device 10 according to the likelihood value of the candidate area having the highest likelihood. That is, the processor 202 causes the imaging device 10 to zoom in or out. Thereafter, the process returns to step S101. In this case, the processor 202 acquires the image of the next frame and repeats the same process.

Figures 9A and 9B are diagrams showing an overview of the control of the angle of view. Here, Figure 9A shows an overview of the control when the likelihood value is high, and Figure 9B shows an overview of the control when the likelihood value is low.

For example, when the maximum likelihood value acquired in the overall likelihood map 700a is greater than a preset threshold, the processor 202 causes the imaging device 10 to zoom in on the coordinates having the maximum likelihood value as shown in FIG. 9A. As a result, in the image 500d captured in the next frame, a narrow range centered on the coordinates having the highest likelihood in the previous frame is enlarged. As a result, it is possible to capture the object O more clearly by enlarging the vicinity of the coordinates having a high probability of being the object O. Also, for example, when the maximum likelihood value acquired in the overall likelihood map 700a is smaller than a preset threshold, the processor 202 causes the imaging device 10 to zoom out on the coordinates having the maximum likelihood value as shown in FIG. 9B. A small likelihood value means that the object including the coordinates is less likely to be the target object O. In this case, the imaging device 10 is zoomed out, so that the surrounding area of the object can be increased. This reduces the possibility that the object O will be lost from the captured image. The likelihood threshold may be set manually.

Returning now to the description of FIG. 6, in step S109, the processor 202 outputs the coordinates of the candidate region having the highest likelihood as the coordinates of the target object O, for example, to a control system of the flying object. In response to this, the control system of the flying object can control the flying object to track the target object O.

As described above, according to the first embodiment, candidate regions of objects are selected by accumulating likelihoods calculated from features extracted in multiple frames. In other words, if a candidate region with high likelihood remains at the same coordinates, the likelihood is high, and if it changes to different coordinates, the likelihood is low. By determining the object based on the likelihood calculated in this way, in the first embodiment, false detections against complex backgrounds are suppressed, and the object can be detected stably and accurately.

(Modification of the first embodiment)
A modified example of the first embodiment will be described. In the first embodiment, the tracking unit 303 calculates the current overall likelihood map by adding the current frame likelihood map to the past overall likelihood map. Alternatively, the tracking unit 303 may calculate the current overall likelihood map by adding a plurality of past frame likelihood maps to the current frame likelihood map.

Figure 10 is a diagram showing an example of feature tracking processing in a modified example of the first embodiment. In the example of Figure 10, an overall likelihood map is calculated from the frame likelihood maps of four frames including the current frame t. The tracking unit 303 calculates frame likelihood maps 600a, 600b, 600c, and 600d from an image 500a of the current frame t, an image 500b of the previous frame t-1, an image 500c of the frame t-1 two frames ago, and an image 500d of the frame t-2 three frames ago. The method of calculating the frame likelihood map may be the same as in the first embodiment. Specifically, the tracking unit 303 calculates the frame likelihood map by adding the likelihood value in each extracted candidate region to the likelihood map of the initial value stored in advance.

The tracking unit 303 of the modified example calculates an overall likelihood map based on the frame likelihood maps of these multiple frames. Specifically, the tracking unit 303 increases the likelihood of a frame likelihood map with a small time difference from the current frame, decreases the likelihood of a frame likelihood map with a large time difference from the current frame, and calculates the overall likelihood map by adding up the likelihood values of each coordinate held by each frame likelihood map.

For example, the tracking unit 303 calculates a frame likelihood map 800a in which the likelihood values of each of the frame likelihood maps 600a for frame t are set to 100%, i.e., the likelihood values are multiplied by a coefficient of 1.0; a frame likelihood map 800b in which the likelihood values of each of the frame likelihood maps 600b for frame t-1 are attenuated to 75%, i.e., the likelihood values are multiplied by a coefficient of 0.75; a frame likelihood map 800c in which the likelihood values of each of the frame likelihood maps 600c for frame t-2 are attenuated to 50%, i.e., the likelihood values are multiplied by a coefficient of 0.5; and a frame likelihood map 800d in which the likelihood of each of the frame likelihood maps 600d for frame t-3 is attenuated to 25%, i.e., the likelihood values are multiplied by a coefficient of 0.25. Then, the tracking unit 303 calculates the overall likelihood map 700a by adding up the likelihoods of the coordinates of the frame likelihood maps 800a, 800b, 800c, and 800d of these four frames. Even with such an overall likelihood map 700a, if there is a likelihood value at the same coordinate in the frame likelihood map of a past frame and the frame likelihood map of the current frame, the overall likelihood map is created so that the likelihood value at that coordinate increases. On the other hand, since the amount of likelihood attenuation is greater for an older frame, the impact of the past likelihood value on the overall likelihood map of the current frame is small. Therefore, even if there is a false detection of an object in a past frame, the impact is corrected over time. Therefore, even in the modified example, it is possible to suppress false detection against a complex background and detect an object stably and accurately.

Figure 11 is a flowchart showing the calculation process of the overall likelihood map in a modified example of the first embodiment. In step S301, the processor 202 obtains a likelihood value from coordinates corresponding to a candidate area in the frame likelihood map in the current frame.

In step S302, the processor 202 attenuates each frame likelihood map by multiplying the likelihood value of each coordinate in the frame likelihood map calculated for the current frame, the previous frame, the frame two frames ago, and the frame three frames ago by a coefficient assigned to each frame.

In step S303, the processor 202 adds the likelihood values of the corresponding coordinates of each attenuated frame likelihood map. In this way, the overall likelihood map for the current frame is calculated.

In step S304, the processor 202 extracts the coordinates and likelihood information of the candidate region with the highest likelihood from the overall likelihood map in the current frame. The processor 202 then ends the processing of FIG. 11.

Even with the above modified example, it is possible to suppress false detections and stably detect objects even against a complex background.

The calculation method of the overall likelihood map shown in the modified example is just one example. For example, the amount of attenuation is not limited to the amount described in the modified example, as long as it is set so that the current frame is the highest. Similarly, the number of frames to be added is not limited to four frames, as long as it includes the current frame. Furthermore, it is not necessary to add the frame likelihood maps of consecutive frames. Furthermore, a past overall likelihood map may be attenuated and used as a past frame likelihood map, as in the first embodiment.

Second Embodiment
Next, a second embodiment will be described. Fig. 12 is a functional block diagram of an object detection device 20 according to this embodiment. In Fig. 12, the same components as those in Fig. 2 are given the same reference numerals as those in Fig. 2. Descriptions of components given the same reference numerals as those in Fig. 12 will be omitted or simplified as necessary.

The object detection device 20 of this embodiment includes an image capture control unit 301, a search range setting unit 306, a feature extraction unit 302, a tracking unit 303, a determination unit 304, and an output control unit 305. For example, the processor 202 operates as the image capture control unit 301, the search range setting unit 306, the feature extraction unit 302, the tracking unit 303, the determination unit 304, and the output control unit 305 by executing the object detection program 2041. The image capture control unit 301, the search range setting unit 306, the feature extraction unit 302, the tracking unit 303, the determination unit 304, and the output control unit 305 may be configured by dedicated hardware or the like.

The search range setting unit 306 sets the position of the search range when the feature extraction unit 302 extracts features. Specifically, the search range setting unit 306 sets the position of the search range in the current frame based on the likelihood value in the overall likelihood map calculated in the past frame.

Figure 13 is a diagram showing an example of the search range setting process of the search range setting unit 306. For example, assume that the likelihood value of the candidate area 701a is the highest in the overall likelihood map 700a calculated from the image 500a obtained in the previous frame. In this case, the search range setting unit 306 sets a range set in advance with, for example, the central coordinate of the candidate area 701a as the center as the search range. Then, the search range setting unit 306 sends the upper left X coordinate X4, the upper left Y coordinate Y4, the width W4, and the height H4 of the set search range to the feature extraction unit 302. Here, the width W4 and the height H4 may be set manually. The feature extraction unit 302 performs feature extraction processing for the search range A in the image 500e of the next frame.

Here, since there is no overall likelihood map for the first frame, the search range setting unit 306 sets the search range to, for example, the entire image.

The search range setting unit 306 may also set multiple search ranges within the image. In this case, the search range setting unit 306 may set multiple search ranges, for example, in descending order of likelihood value in the overall likelihood map. In this case, the search range setting unit 306 may set the multiple search ranges so that they do not overlap with each other.

The search range setting unit 306 may also set the size of the search range, i.e., the width W4 and height H4, according to the highest likelihood value in the overall likelihood map calculated in the previous frame.

Figures 14A and 14B are diagrams showing modified examples of the search range setting process. Here, Figure 14A is a diagram showing a modified example of the search range setting process when the likelihood value is low. Also, Figure 14B is a diagram showing a modified example of the search range setting process when the likelihood value is high.

For example, if the maximum likelihood value in the overall likelihood map calculated in the previous frame is lower than a preset threshold, the search range setting unit 306 sets a search range A1 that is expanded by a certain width and height from the original search range with the highest likelihood as the center as shown in FIG. 14A, and sends the upper left X coordinate X5, the upper left Y coordinate Y5, the width W5, and the height H5 of the search range A1 to the feature extraction unit 302. On the other hand, if the maximum likelihood value in the overall likelihood map calculated in the previous frame is higher than a preset threshold, the search range setting unit 306 sets a search range A2 that is reduced by a certain width and height from the original search range with the highest likelihood as the center as shown in FIG. 14B, and sends the upper left X coordinate X6, the upper left Y coordinate Y6, the width W6, and the height H6 of the search range A2 to the feature extraction unit 302. Here, the threshold may be one or more. When there are multiple thresholds, the width and height of the search range may be set according to the thresholds.

Figure 15 is a flowchart showing an example of processing of the object detection device 20 in the second embodiment. The processing in Figure 15 is performed by, for example, the processor 202. Here, in Figure 15, the explanation of processing similar to that in Figure 6 is appropriately omitted or simplified.

In step S401, the processor 202 acquires an image from the imaging device 10.

In step S402, the processor 202 sets a search range for the acquired image. As described above, the search range for the first frame may be the entire image. On the other hand, the search range after the first frame is set based on the overall likelihood map calculated in the previous frame. The processor 202 sets the search range in a range centered on the coordinate having the highest likelihood value in the overall likelihood map calculated in the previous frame. At this time, the processor 202 may set the width and height of the search range according to a comparison between the highest likelihood value and a threshold value.

The processing in steps S403-S410 is the same as the processing in steps S102-S109 in FIG. 6. Therefore, the description is omitted.

As described above, in the second embodiment, the search range for the features of the next frame is set based on the likelihood value of the overall likelihood map calculated in the previous frame. As mentioned above, the object may not move significantly between frames. In this case, by limiting the search range in the image, erroneous extraction of the background, etc. can be suppressed.

[Other Modifications]
The object detection program executed in the object detection device 20 of each of the above-described embodiments and modified examples thereof may be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD). The object detection program may also be stored on a computer connected to a network such as the Internet, and the object detection device 20 may download and use the object detection program via the network as necessary.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

1 Object detection system, 10 Imaging device, 20 Object detection device, 201 Input device, 202 Processor, 203 Memory, 204 Storage device, 205 Output device, 301 Imaging control unit, 302 Feature extraction unit, 303 Tracking unit, 304 Determination unit, 305 Output control unit, 306 Search range setting unit.

Claims (13)

a feature extraction unit that extracts candidate regions of an object in an image acquired in time series from an imaging device and likelihoods of the candidate regions based on feature amounts of the images;
a tracking unit that tracks a candidate area that is highly likely to be the object from among the candidate areas in the current image by accumulating the likelihoods extracted in time series;
a determination unit that determines a final position of the object based on the candidate area tracked by the tracking unit;
Equipped with
The tracking unit is
adding a likelihood value in a second candidate region extracted based on the current image to a frame likelihood map having a likelihood value in a first candidate region extracted based on a past image;
subtracting a constant value from the likelihood value at coordinates of the frame likelihood map excluding the second candidate region;
An object detection device that performs tracking based on a total likelihood map calculated by the addition and subtraction . a feature extraction unit that extracts candidate regions of an object in an image and likelihoods of the candidate regions based on feature amounts of the images acquired in time series from an imaging device;
a tracking unit that tracks a candidate area that is highly likely to be the object from among the candidate areas in the current image by accumulating the likelihoods extracted in time series;
a determination unit that determines a final position of the object based on the candidate area tracked by the tracking unit;
Equipped with
The tracking unit is
adding a second frame likelihood map having a likelihood value in a second candidate region extracted based on a current image to a first frame likelihood map having a likelihood value in a first candidate region extracted based on a past image;
Tracking based on the overall likelihood map calculated by the addition;
An object detection apparatus that attenuates the first frame likelihood map in accordance with a time difference between the first frame likelihood map and the second frame likelihood map during the addition. The object detection device according to claim 1 or 2, wherein the feature extraction unit deletes, from the extracted candidate regions, candidate regions in which at least one of the likelihood and the luminance value of a central region of the candidate region is lower than a certain value. The object detection device according to claim 3 , wherein the feature extraction unit sets the constant value based on a difference between average luminance values of a central region and a peripheral region of the candidate region. a search range setting unit that sets a search range of the image in which the extraction by the feature amount extraction unit is performed,
The object detection device according to claim 1 , wherein the search range setting unit sets the position of the search range based on an accumulation result of the likelihood. The object detection device according to claim 5 , wherein the search range setting unit further sets a size of the search range based on an accumulation result of the likelihood. The object detection device according to claim 1, further comprising an imaging control unit that controls at least one of the imaging direction and the angle of view of the imaging device that captures the image based on the accumulation result of the likelihood. The object detection device according to claim 1, which is mounted on a flying object. The object detection device according to claim 8 , further comprising an output control unit that outputs information on the position of the object determined by the determination unit to a control system of the flying object. extracting a candidate region of an object in an image acquired in time series from an imaging device and a likelihood of the candidate region based on feature amounts of the image;
Tracking a candidate area that is highly likely to be the object from among the candidate areas in the current image by accumulating the likelihood extracted in time series; and
determining a final object position based on the tracked candidate regions; and
Equipped with
Tracking a candidate region that is likely to be an object of interest includes:
adding a likelihood value for a second candidate region extracted based on the current image to a frame likelihood map having a likelihood value for a first candidate region extracted based on a past image;
subtracting a constant value from the likelihood value at coordinates of the frame likelihood map excluding the second candidate region;
tracking based on a total likelihood map calculated by the addition and the subtraction;
An object detection method comprising : extracting a candidate region of an object in an image acquired in time series from an imaging device and a likelihood of the candidate region based on feature amounts of the image;
Tracking a candidate area that is highly likely to be the object from among the candidate areas in the current image by accumulating the likelihood extracted in time series; and
determining a final object position based on the tracked candidate regions; and
Equipped with
Tracking a candidate region that is likely to be an object of interest includes:
adding a second frame likelihood map having a likelihood value in a second candidate region extracted based on a current image to a first frame likelihood map having a likelihood value in a first candidate region extracted based on a past image;
tracking based on the overall likelihood map calculated by the addition;
attenuating the first frame likelihood map according to a time difference between the first frame likelihood map and the second frame likelihood map during the addition;
An object detection method comprising: extracting a candidate region of an object in an image acquired in time series from an imaging device and a likelihood of the candidate region based on feature amounts of the image;
Tracking a candidate area that is highly likely to be the object from among the candidate areas in the current image by accumulating the likelihood extracted in time series; and
determining a final object position based on the tracked candidate regions; and
An object detection program for causing a processor to execute the above,
Tracking a candidate region that is likely to be an object of interest includes:
adding a likelihood value for a second candidate region extracted based on the current image to a frame likelihood map having a likelihood value for a first candidate region extracted based on a past image;
subtracting a constant value from the likelihood value at coordinates of the frame likelihood map excluding the second candidate region;
tracking based on a total likelihood map calculated by the addition and the subtraction;
An object detection program including : A candidate for an object in an image is selected based on a feature amount of the image acquired in time series from an imaging device.
extracting likelihoods of the complement region and the candidate region;
The likelihoods extracted in time series are accumulated to obtain a candidate region in the current image.
tracking a candidate region that is highly likely to be the object from among the candidate regions;
determining a final object position based on the tracked candidate regions; and
An object detection program for causing a processor to execute the above,
Tracking a candidate region that is likely to be an object of interest includes:
adding a second frame likelihood map having a likelihood value in a second candidate region extracted based on a current image to a first frame likelihood map having a likelihood value in a first candidate region extracted based on a past image;
tracking based on the overall likelihood map calculated by the addition;
attenuating the first frame likelihood map according to a time difference between the first frame likelihood map and the second frame likelihood map during the addition;
An object detection program including: