JP7208610B2 - Vehicle recognition system and vehicle recognition method - Google Patents

Description

The present invention relates to a vehicle recognition apparatus and a vehicle recognition method for recognizing and extracting a vehicle from captured images.

Japan is prone to natural disasters such as earthquakes, typhoons, and volcanoes. These disaster sites require immediate recovery work, but they are too dangerous for people to enter. Therefore, currently, unmanned construction technology is being developed and introduced, in which heavy machinery is remotely operated from a safe location to perform restoration work. Heavy equipment operators rely on the images from surveillance cameras installed on site to have heavy equipment perform construction work. However, at present, a camera operator remotely controls the direction of the surveillance camera in order to allow the surveillance camera to capture images of heavy machinery working in a wide range.

In order to reduce the work of the camera operator and increase efficiency, there is a demand for a tracking camera that automatically tracks the heavy machinery. Some heavy machinery is equipped with a high-precision GPS, and it is technically possible to have the camera automatically follow the heavy machinery in conjunction with the GPS. However, in the actual field, it is difficult to equip all heavy equipment with GPS.

Masato Fujitake and Takashi Yoshimi ”Estimation System of Construction Equipment from Field Image by Combination Learning of its Parts” 2017 11th Asian Control Conference (ASCC) Gold Coast Convention Centre, Australia December 17-20, 2017, pp.1672-1676

As a technique for recognizing heavy machinery from captured images, there is a vehicle recognition method that performs learning using a neural network and extracts heavy machinery from captured images. However, in vehicle recognition using conventional neural networks, there are cases where heavy machinery cannot be recognized when part of the machinery is hidden or when the orientation of the machinery has changed. A described vehicle recognition method is proposed. However, the vehicle recognition by the neural network still does not have high recognition accuracy, and there are problems such as erroneously recognizing another vehicle existing at the site.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a vehicle recognition system and a vehicle recognition method that can stably follow heavy machinery based on surveillance camera images.

A vehicle recognition system according to an aspect of the present invention includes an imaging unit that captures an image including heavy machinery, and a template image that includes the heavy machinery and an image captured by the imaging unit are compared by template matching to determine the movement of the heavy machinery. and extracting an image area including the heavy machinery from the captured image, the image processing unit estimating the position of the extracted image area on the image and extracting The image area is updated as a new template image, and when the next template matching is performed based on the updated template image and the new image captured by the imaging unit, in the new image, from the position: An image region including the heavy machinery is extracted within a range of a predetermined distance.

A vehicle recognition system according to an aspect of the present invention includes an imaging unit that captures an image including heavy machinery, and a template image that includes the heavy machinery and an image captured by the imaging unit, which are compared by template matching. A first image processing unit that extracts a first image region including the heavy machinery from an image; and a plurality of parts that configure the heavy machinery by applying deep learning results by a neural network based on the image captured by the imaging unit. separately, recognizes the overall shape of the heavy machinery by combining the recognition results of the plurality of parts, and extracts a second image region including the heavy machinery, wherein the first image The processing unit acquires information of the second image area from the second image processing unit, compares the first image area and the second image area, and if the comparison result satisfies a criterion, the second image area A region is determined as a recognition result of the heavy equipment.

A vehicle recognition system according to an aspect of the present invention includes an imaging unit that captures an image including heavy machinery, and a template image that includes the heavy machinery and an image captured by the imaging unit, which are compared by template matching. A first image processing unit that extracts a first image region including the heavy machinery from an image; and a plurality of parts that configure the heavy machinery by applying deep learning results by a neural network based on the image captured by the imaging unit. separately, recognizes the overall shape of the heavy machinery by combining the recognition results of the plurality of parts, and extracts a second image region including the heavy machinery, wherein the first image The processing unit acquires information about the position of the second image region from the second image processing unit, extracts a third image region including the heavy machinery in the image based on the position information from the image, and extracts the third image region. updating the third image region as a new template image, performing the following template matching based on the updated template image and the image newly captured by the imaging unit, and extracting the image region including the heavy machinery. .

In a vehicle recognition method according to an aspect of the present invention, a computer captures an image including heavy machinery, compares a template image including the heavy machinery with the captured image by template matching, and determines movement of the heavy machinery from the captured image. is detected, an image region including the heavy machinery is extracted, a position of the extracted image region on the image is estimated, the extracted image region is updated as a new template image, and the updated template image and the captured When performing the next template matching based on the new image thus obtained, an image region including the heavy machinery is extracted from the new image within a range of a predetermined distance from the position.

In a vehicle recognition method according to an aspect of the present invention, a computer captures an image including heavy machinery, compares a template image including the heavy machinery with the captured image by template matching, and determines whether the captured image includes the heavy machinery. A first image region is extracted, based on the imaged image, a deep learning result by a neural network is applied to individually recognize a plurality of parts constituting the heavy machinery, and the recognition results of the plurality of parts are combined. recognizing the overall shape of the heavy machinery, extracting a second image region including the heavy machinery, obtaining information of the second image region, comparing the first image region and the second image region, and obtaining a comparison result. satisfies the criterion, the second image region is determined as the recognition result of the heavy equipment.

In a vehicle recognition method according to an aspect of the present invention, a computer captures an image including heavy machinery, compares a template image including the heavy machinery with the captured image by template matching, and determines whether the captured image includes the heavy machinery. A first image region is extracted, based on the imaged image, a deep learning result by a neural network is applied to individually recognize a plurality of parts constituting the heavy machinery, and the recognition results of the plurality of parts are combined. recognizing the overall shape of the heavy machinery, extracting a second image region including the heavy machinery, obtaining information about the position of the second image region, and recognizing the heavy machinery in the image based on the positional information from the image. extracting a third image region including the heavy equipment, updating the extracted third image region as a new template image, performing the following template matching based on the updated template image and the newly captured image, and including the heavy equipment Extract image regions.

According to the present invention, heavy machinery can be stably tracked based on surveillance camera images.

It is a figure which shows the state which monitors an unmanned heavy machine with a camera at a work site. 1 is a block diagram showing an example of the configuration of a vehicle recognition system; FIG. FIG. 10 is a diagram showing an example of template matching using a template image; FIG. 10 is a diagram showing changes in the size of a detection target for template matching; FIG. 10 is a diagram showing a change in posture of a target to be detected by template matching; It is a figure which shows an example which extracts a heavy machine based on image processing. It is a figure which shows an example of labeling processing. FIG. 4 is a conceptual diagram of IOU calculation; It is a figure which shows the state by which some heavy machines were concealed. FIG. 11 is a conceptual diagram showing an example of the contents of processing of the second method; It is a figure which shows an example of the process result of a 2nd method. FIG. 10 is a diagram showing an example of bucket detection results using different patterns of test data after machine learning training. 4 is a sequence diagram showing an example of the flow of processing executed in the vehicle recognition system; FIG. It is a figure which shows an example of the operation result of a vehicle recognition system. It is a figure which shows an example of the operation result of a vehicle recognition system. It is a detection result for each fixed frame when tracking is performed by a general tracking method using an evaluation video. It is a detection result for each fixed frame when tracking is performed only by the first method in the evaluation video. It is a detection result for each fixed frame when following is performed by processing of the vehicle recognition system using the video for evaluation. It is a figure which shows an example of the evaluation result of the detection rate of heavy machinery based on each detection method. The detection results of the embodiment and its modification are compared. It is a figure which shows the detection result of embodiment and its modification. It is a figure which shows an example of the evaluation result of embodiment and its modification.

An embodiment of the vehicle recognition device of the present invention will be described below with reference to the drawings.

As shown in FIG. 1, the vehicle recognition system 1 is a system that uses a camera to track a vehicle (for example, a heavy machine) that automatically performs work at a work site.

FIG. 2 is a block diagram showing an example of the configuration of the vehicle recognition system 1. As shown in FIG. The vehicle recognition system 1 includes, for example, an imaging unit 10 including a camera that captures the situation of the site, and a first recognition device 20 ( and a second recognition device 30 (second image processing unit) that processes an image captured by the imaging unit 10 based on a second method and recognizes the vehicle.

The imaging unit 10 includes a controllable pan head, and is controlled by the first recognition device 20 so that the heavy machinery enters the imaging area. The first recognition device 20 is, for example, a terminal device such as a PC (Personal Computer) placed on site. The first recognition device 20 and the imaging unit 10 are communicably connected by wire or wirelessly. The first recognition device 20 is, for example, installed at the site together with the image capturing unit 10 and acquires image data captured by the image capturing unit 10 . The first recognition device 20 controls the imaging unit 10 using a first method (ATM3D: Automated Template Matching meets motion detection) described later, in which template matching is constantly improved based on the image captured by the imaging unit 10. to follow heavy machinery.

The first recognition device 20 can, for example, communicate with the second recognition device 30 via the network NW to send and receive data. The second recognition device is, for example, a terminal device that has a GPU (Graphics Processing Unit) and can execute processing faster than the first recognition device 20 . The second recognition device 30 recognizes heavy machinery using a second method (POLO: Parts-based YOLO (You Only Look Once)) based on deep learning by a neural network based on the image data acquired from the first recognition device 20. conduct.

The vehicle recognition system 1 is a tracking system (ATCS: Automatic Tracking Camera System) that realizes robust camera tracking of heavy machinery by integrating the above two detection methods based on image processing.

First, the first recognition device 20 acquires an image including heavy machinery captured by the imaging unit 10 via a LAN (Local Area Network) cable. The first recognition device 20 recognizes the heavy machinery using the first method based on the image including the heavy machinery. At the same time, the first recognition device 20 transmits the image of the heavy machinery to the second recognition device 30 via the network NW. The second recognition device 30 recognizes the heavy machinery by the second method based on the acquired image including the heavy machinery.

The first recognition device 20 performs recognition processing of heavy machinery using the first method until it receives the recognition result from the second recognition device 30, and the first recognition device 20 captures an image based on the recognition result. Information including a control signal for controlling the camera is transmitted to the unit 10 to cause the camera to follow the heavy machinery.

Further, when the first recognition device 20 receives the recognition result from the second recognition device 30, the first recognition device 20 integrates the result of the first method and the result of the second method, recognizes the position of the heavy machinery on the image, and obtains the position of the heavy machine. Based on the final result obtained, the camera is controlled to follow the heavy machinery. By repeating this, the vehicle recognition system 1 can perform real-time and robust follow-up control. Next, the details of each of the first method (ATM3D) and the second method (POLO) will be described.

First, the processing of the first method (ATM3D) executed in the first recognition device 20 will be described. ATM3D is a camera tracking method that combines template matching-based tracking and motion detection. ATM3D is good at continuously detecting heavy machinery in the camera image while the camera is moving.

As shown in FIG. 3, template matching is one method of searching for an object within an image. Template matching is a method of extracting an image region (first image region) having the highest degree of similarity to a pre-prepared image (template image) smaller than the captured image, from the captured image. The template image is, for example, an image of heavy machinery. As for the template image, for example, images of a plurality of patterns created in advance are stored in the storage unit of the first recognition device 20 . The first recognition device 20 can follow the heavy machinery on the captured image by continuously performing template matching in real time.

However, although template matching can stably track an object when the posture of the heavy machinery does not change, there are situations where the appearance of the heavy machinery changes significantly, such as when the orientation of the heavy machinery changes or part of its appearance is hidden. Then you may not be able to follow the heavy machinery. Therefore, the template image used in template matching is always automatically updated by the method described below to improve robustness.

First, the first template image is selected by the user performing a mouse drag operation on the display screen of the first recognition device 20 to specify a rectangular area. The first recognition device 20 compares the selected first template image with the next frame image captured by the camera, performs template matching, and detects the image region with the highest degree of similarity from the frame image. . Then, the first recognition device 20 estimates the position of the heavy machinery on the image based on the extracted image area. Next, the first recognition device 20 updates the image area extracted from the frame image as a new template image.

When the camera captures a new frame image, the first recognition device 20 compares the updated template image with the new frame image, extracts an image region including the heavy machinery from the new frame image, and extracts the heavy machinery. Along with tracking, the extracted image area is updated as a new template image. After that, the first recognition device 20 follows the heavy machinery while repeating these processes.

When performing new template matching, the first recognition device 20 searches for heavy machinery within a predetermined distance from the immediately preceding detection position in order to speed up processing. This is because it is impossible for the heavy equipment to move greatly within the screen during the time of one frame, and there is no need to search the entire screen.

However, according to this processing method, since there is no function to change the size of the template image, as shown in FIG. It may become impossible to follow.

In addition, when a movement such as a large rotation of the heavy machinery occurs from the initial template image, the surface that was visible as shown in FIG. .

Therefore, in order to further improve the robustness of following heavy machinery, we focus on the "movement" of heavy machinery and supplement template matching using background subtraction, which is a method of motion detection. Background subtraction is to compare two images and extract the difference, and extract changes by comparing the image of the currently captured frame of the camera video with the image of the frame captured in the past. . That is, when the camera is not moving, moving objects can be extracted, so it can be used as "motion detection".

The reason why template matching cannot follow up is that the visible surface disappears due to the rotation of the heavy machinery. However, when the heavy machinery turns and the visible surface disappears, it means that a large change (movement) has occurred. is. Therefore, the first recognition device 20 supplements motions such as turning of heavy machinery, which are difficult to detect by template matching, by motion detection based on background subtraction.

However, with background subtraction alone, when the camera moves or zooms, the entire screen changes, so it may be difficult to detect only moving objects. FIG. 6 is a diagram showing an example of extracting heavy machinery based on image processing.

FIG. 6(b) shows the result of detecting a moving heavy machine using the background difference in FIG. 6(a). In the binarized image obtained by capturing the image of the heavy machinery only in black and white, the white portion is the portion where the change is extracted. As shown in FIG. 6B, when the background subtraction is performed by comparing the preceding and succeeding frame images, only the frame of the moving object may be extracted. A technique for improving the background difference recognition rate will be described below.

As shown in FIG. 6(c), for example, background difference data obtained in several frames captured in the past are superimposed to leave a predetermined position of the heavy machine like an afterimage, as shown in FIG. 6(d). You can emphasize the moving part of the heavy equipment so that it can be seen. Further, if the extracted area is expanded, the moving object can be clearly extracted as shown in FIG. 6(e).

However, the moving object is not recognized as a single mass at this point. The first recognition device 20 performs image processing by labeling, which will be described later, for recognizing the connected portion as one object, and extracts it as a moving object. After extracting the moving object, the first recognition device 20 draws a rectangle (bounding box) surrounding the object on the image so that the mass of the moving object is inscribed (see FIG. 6F).

Next, the principle of labeling will be explained. As shown in FIG. 7, labeling is image processing performed on a binarized image. The first recognition device 20 assigns the same label (number) to all pixels belonging to the same block (connected component) in the binarized image, and assigns different labels to different connected components.

As shown, the black squares represent pixels and the white areas are adjacent consecutive pixels with the same number. The first recognition device 20 performs such processing to extract clusters included in the binarized image. By applying labeling to the binarized image obtained by the background subtraction in this way, the objects in the image are segmented. Then, the position of the heavy machinery on the image is detected by drawing an inscribed rectangle with respect to the area of the object.

However, the labeled object does not necessarily have the same number in the next frame. In the present embodiment, labeling is performed in descending order of size. In particular, data with a small size are often meaningless data such as noise, and are not to be tracked. Also, since the tracked target does not necessarily have the maximum area, it should be noted that multiple objects including the tracked target may be detected.

Background subtraction can detect a moving object on an image, but when the camera moves, the entire image is recognized as a moving object, so heavy machinery alone cannot be tracked. Also, when the object to be tracked stops and another object moves, there is a possibility that the object that has moved will be the object to be tracked.

Therefore, the first recognition device 20 mainly performs tracking using “template matching” in recognition processing of heavy machinery, and when detecting a movement such as a large posture change of the heavy machinery, “motion detection” based on background difference is performed. is used to compensate for template misdetection. "Follow by Template Matching" results in the location containing the rectangular image that is most similar to the template. Also, in "motion detection", a plurality of rectangular images inscribed in a plurality of clusters obtained from labeling are obtained.

Here, if there is a rectangular image obtained by "motion detection" sufficiently close to the rectangular image obtained by "following by template matching", that rectangular image is used as a new template image. Conversely, if there is no "motion detection" rectangular image close to the rectangular image obtained by "template matching follow-up", follow-up is continued using the result rectangle obtained by "template matching follow-up". Also, the degree of similarity between two result rectangles is determined using IoU (Intersection over Union) shown below.

As shown in FIG. 8, IoU is a quantification of the degree of overlapping of multiple rectangular regions, and the area of intersection of multiple rectangular regions is the area covered by the multiple rectangular regions. It can be calculated by dividing by the area of the union. A calculation formula for obtaining the IoU value is as shown in formula (1).
IoU＝(Area of intersection)/(Area of union) (1)
In other words, the closer the IoU value is to the maximum value of "1", the higher the degree of similarity between the positions and sizes of the rectangles.

Next, the mechanism of the second method (POLO) executed by the second recognition device 30 will be described. POLO is an object detection method that detects an object from the results of deep learning based on neural network-based machine learning, and can derive recognition results that are higher than the template matching and background subtraction described above. In object detection using machine learning, it is possible to recognize even if the appearance of the target is slightly different, and to be able to detect the detection target at any size during detection, such as template matching. It is possible to solve problems that could not be realized by conventional image processing methods.

However, processing results using POLO require computer processing time and are difficult to use for detecting heavy machinery in real time. Therefore, by using the recognition result of the second method of the second recognition device 30 to complement the first method of the first recognition device 20, the robustness of heavy machinery recognition in the vehicle recognition system 1 is improved.

Here, the neural network is an animal neural network artificially reproduced by an algorithm on a computer. A neural network performs learning and outputs an output result obtained by applying the learning result to input data. Here, learning means updating a plurality of parameters of the evaluation function used for calculation through iterative training based on data provided for learning. After learning, when observation data obtained by observation is input, calculation is performed based on the evaluation function to which the parameter based on the learning result is applied, and the output value that reflects the learning result is output. be.

Currently, machine learning is being researched to improve the accuracy of algorithms for more accurate detection of objects and the positional accuracy of detected objects. Although these algorithms based on neural networks get better at detecting when there is enough data to train them, problems arise in detection when applied to unmanned construction site surveillance camera tasks. For example, a heavy machine, which is an object to be detected, is sometimes hidden by embankments or trees, making it impossible to capture the entire object, making it difficult to detect the object.

As shown in FIG. 9, when a portion of the heavy machinery is hidden by an obstacle, the entire heavy machinery cannot be captured, and recognition of the heavy machinery by the object detection algorithm is hindered. This is because the detected object looks different from the object learned based on the data. Such a situation occurs while the object is moving to its destination.

In order to realize automatic tracking by the camera, it is necessary to detect heavy machinery more robustly even in such a state. It is conceivable to prepare learning data of hidden heavy machinery and detect heavy machinery by learning, but the way heavy machinery hides differs depending on the situation, so we collect data that records all the appearances of heavy machinery. is difficult.

However, humans can recognize the object if they can see a partial area of the object as shown in FIG. 9, except for the situation where the object cannot be seen completely. This is because, based on the information of the visually recognized area, humans apply known knowledge to infer the invisible part and complement the information.

For example, as shown in FIG. 9(a), the object is assumed to be a crawler dump because the crawler portion and part of the loading section of the dump truck can be seen through the trees. Therefore, the inventors have found that by learning what parts (parts) an object is composed of, it is possible to detect a heavy machine even in a situation where a part of the machine is hidden. This is different from existing tracking methods based on convolutional networks, which only consider the entire appearance of the object.

The proposed object detection method based on the parts of the object recognizes multiple parts that make up the heavy machinery of the object individually, combines the recognition results of the multiple parts to recognize the overall shape of the heavy machinery, and recognizes the type of heavy machinery and the shape of the heavy machinery. Estimate the position on the image.

According to the method proposed by the inventors, by focusing on the parts of the heavy machinery, it is possible to give a characteristic that it is strong in recognition when a part of the heavy machinery is hidden. You can also give it the property of being strong. Changes in the appearance of parts are often small compared to changes in the overall appearance caused by rotation and work. Therefore, by combining the information of the parts, it becomes easier to detect even when the appearance of the whole changes drastically.

Hereinafter, the process of the second method will be described in which the parts of the heavy equipment when the heavy equipment is hidden are focused on and combined to estimate the whole. In the process of the second method, the current object detection model is used as the backbone for the part detection process. As a backbone, for example, an image object recognition algorithm (eg, YOLOv2) by deep learning using a convolutional neural network is used.

This object recognition algorithm is one of the mainstream object detection methods using machine learning, and it is said that high-speed processing is possible for real time while maintaining detection accuracy comparable to other algorithms. It has characteristics.

Convolutional neural networks usually require more computations than conventional image processing techniques. This automatic tracking system requires real-time processing of moving images, so it is important that the detection accuracy is high, but it is also important that the calculation speed is fast. A YOLO model focused on a part is hereinafter referred to as Part based YOLO (POLO).

Unlike other machine learning based object detection methods, YOLOv2 is entirely constructed with convolutional neural networks.

The calculation result after convolution in YOLOv2 processing is a tensor shape, and information such as what kind of category is included in the target area in YOLOv2, how much reliability it has, and where the coordinate position is. Learning is done to be included.

In the process of YOLOv2, noise-like results are filtered, and the calculation results are arranged and output by post-processing. If YOLOv2 is trained to detect not only the entire heavy machine but also the whole heavy machine, the tensor after convolution of YOLOv2 will be output with a lot of information related to part detection. Therefore, using these calculation results, a detection model that combines parts and infers the whole is constructed by attaching a network that makes an inference to detect the whole after the processing stage of YOLOv2.

FIG. 10 is a conceptual diagram showing an example of the contents of the processing of the second method. A box-shaped object labeled 32×32 on the left shows the output result of YOLOv2 expressing the tensor of the feature map. 32×32 is the size of the feature map. The thickness of the feature quantity map and the number of channels are determined by the following equation (2).

channels = num × (classes + coords + 1) (2)

In equation (2), num is the estimated number per grid (eg, 5). classes are estimated classes, and coords is the number of coordinate parameters to be estimated (eg, 4).

In FIG. 10, between the feature map on the left and the feature map on the right is processing for inferring the overall shape of the heavy machinery by combining information on the recognized parts of the heavy machinery. The feature amount map on the right is the feature amount map after conversion, and the position and type of the entire heavy machinery on the image are estimated using this output value. The estimation from the feature quantity map uses the same estimation as in YOLOv2.

Training data sets (learning data) and evaluation data sets for detecting heavy machinery by POLO are video data captured at unmanned construction sites and heavy machinery made with simulated RC (Radio Control) models. It was created using the video that was taken. Image regions containing heavy machinery were extracted by randomly extracting frames from the video.

In this dataset, the types of heavy equipment are backhoes and dumps (including crawlers). The format of the data set is the format of the VOC data set, and the categories of the dump data set include tires, dumping beds, taxi parts (parts), and the shape of the entire dump. Backhoe datasets also include the full backhoe geometry, including arms, booms, buckets, counterweights, shoes, and cabs. Approximately 2000 images containing dumps and 1500 images of crawler dumps were prepared and used in a 4:1 split for training and testing.

As shown in FIG. 11, in the image area containing the heavy machinery of the data set, the image area surrounded by a small frame indicates the part of the heavy machinery, and the image area surrounded by a large frame (second image area) is the entire heavy machinery. indicates

Next, a POLO training method will be described. In general, machine learning using neural networks with many parameters is known to cause overfitting, in which the training data is overfitted when learning is performed with a small number of data. Transfer learning is a solution to this problem. This is a method of learning parameters in advance using a large data set of another task when there is little data for a certain task, and then learning the original task with a small data set.

By using transfer learning, the accuracy is better than the method of learning randomly initialized parameters from 0. Object detection datasets are more difficult to create teacher data than image recognition datasets. The object recognition data set specifies the corresponding category label for a certain image. This is because it is necessary to specify coordinates and their categories.

Therefore, creating an object detection dataset takes time and effort, and the number of datasets required for image recognition training is small, so when training an object detection model using machine learning, , using a dataset prepared for object recognition, learn parameters that are easy to obtain good feature values. As a result of learning, after parameters that can be used as a model for object detection are generated and the parameters are initialized, learning is performed with a data set for object detection as transfer learning.

The second recognition device 30 uses the transfer learning described above in the heavy machinery recognition process. The POLO process performed in the second recognizer 30 has two steps. The first stage is a training process in which multiple parts of the heavy equipment are detected, including the overall shape of the heavy equipment. The second step is a step of learning a converter that converts the feature map obtained by the model learned in the first step, integrates a plurality of parts, and estimates the overall shape of the heavy equipment.

The first stage training process used a similar setup as the YOLOv2 training. The different part is the number of classes to be detected, and the detection target is the part and the whole of the heavy machinery. The reason why the detection of the entire heavy machinery is included in the first stage of learning is that if it is already possible to detect the entire heavy machinery in the first stage, that information will be used, and if it is not possible to detect the whole, it will be combined. It is for

In the second stage training process, the parameters trained in the first stage are fixed and only the parameters of the newly added transducer are learned. There are only two types of labels used for learning: the entire backhoe and the entire crawler dump, without using the training data of the parts.

In the dataset above, it was confirmed that it is possible to detect heavy equipment in most conditions. As for the state that could not be detected, it is considered that the bucket is in a state where it is difficult to see or it is small and difficult to detect. The training dataset used this time is smaller than the dataset used to train normal object detection, so in order to improve accuracy, it is necessary to increase the size of the dataset and make it easier to obtain feature values.

When following heavy machinery, it is necessary to detect the entire heavy machinery, but it is assumed that attention will be paid to the bucket during work such as a backhoe that operates the bucket by arm movement. Therefore, after completing the learning of the two-step process for the bucket, it was confirmed whether or not the part information about the bucket could be extracted from the intermediate result of the calculation processing and detected.

Since the final processing result of POLO itself is the overall estimation, in order to extract the part information, it is necessary to use the result of YOLOv2, which is the result before the part information integration inference part, and reason about the part information. Since the parameters used in the first-stage process are fixed even after the two-stage process training, it is possible to make inferences about the site information. FIG. 12 is a diagram illustrating an example of bucket detection results using different patterns of test data after machine learning training.

Next, a method for integrating the above two detection results of ATM3D and POLO will be described. As described above, after the initial template is set, the ATM3D has a higher follow-up accuracy as the amount of change decreases, but may not be able to follow if the amount of change increases.

On the other hand, POLO does not use time-series data and therefore does not depend on the amount of change, but there is a possibility of detecting other heavy machinery or of suddenly erroneous detection. On the other hand, the detection result of ATM3D rarely deviates suddenly from the object to be tracked. Therefore, the POLO detection result is adopted only when the POLO detection result is similar to the ATM3D detection result.

For example, ATM3D and POLO are both represented by a rectangle surrounding the heavy machinery on the screen as the detection result of heavy machinery. decide whether to adopt. After that, regardless of which one is adopted, the camera platform is controlled and the template image used in ATM3D is updated according to the adopted rectangular position. However, since POLO performs processing on a computer equipped with a GPU, it is assumed that an online external computer will be used.

In the field, the first recognition device 20 communicates with the second recognition device 30, and after transmitting the image captured by the imaging unit 10, the second recognition device 30 performs detection processing and sends the detection result to the first recognition device 20. It takes time based on the communication environment until the For example, in the experimental example, it took about 0.25 to 1 second per frame from the start of image transmission to the return of detection results. In the experimental example, camera tracking by ATM3D and detection processing by POLO were processed in parallel in order to prevent providing an operator with a low frame rate (discontinuous) video.

Next, a camera control method will be described. Instead of constantly moving the camera, it is necessary to move the camera as large as possible so that the heavy machinery does not protrude from the screen. Therefore, the first recognition device 20 sets a rectangular area of a certain size in the center of the screen, and operates the camera so that the tracking target can fit inside the rectangular area. First, the first recognition device 20 acquires the position of the rectangular area surrounding the heavy machinery based on the detection result.

Next, the first recognition device 20 operates the camera pan head at a constant speed so that the central position of the detected rectangular area is within the rectangular central area of a predetermined size including the center of the screen. When the center of the rectangular area of the heavy equipment falls within the range, the camera pan head is stopped. For example, when the heavy machinery is in the upper left corner of the screen, the first recognition device 20 continues to operate the camera pan head so that the rectangular area of the heavy machinery fits within the central area of the screen. The first recognition device 20 stops the rotation of the camera pan head when the center of the rectangular area of the heavy equipment fits within the center area. As a result, the heavy machinery is displayed near the center on the screen of the display unit of the first recognition device 20 .

The size of the rectangular center area displayed on the screen is set to an arbitrary size by the operator's operation. When the size of the central area is set small, the first recognition device 20 frequently operates the camera pan head in response to the movement of the heavy equipment so that the center of the rectangular area of the heavy equipment is contained within the central area. However, the heavy equipment will be displayed in the center of the screen.

If the center area is set too small, when the moving detection target enters the center area, before the camera pan head moves and stops, the pan head moves further in accordance with the movement of the detection target, causing It may stick out, and depending on the case, the camera pan head may vibrate. This phenomenon particularly depends on the delay time until the camera image is acquired by the first recognition device 20 and the speed of the pan head motor.

On the other hand, when the size of the central region is set large, the first recognition device 20 does not operate the camera pan head even if the heavy equipment is slightly moved. In the experiment, we empirically set up a rectangular area in which tracking can be performed, which is thought to be appropriate for the operation of heavy machinery.

Next, the flow of processing executed in the vehicle recognition system 1 will be described. FIG. 13 is a sequence diagram showing an example of the flow of processing executed in the vehicle recognition system 1. As shown in FIG.

First, the first recognition device 20 acquires an image of one frame from the imaging unit 10 (step S100). The first recognition device 20 simultaneously transmits this image data to the second recognition device 30 . The first recognition device 20 performs follow-up processing of the heavy machinery by processing using ATM3D based on the image acquired from the imaging unit 10 (step S102). The first recognition device 20 repeats the processing of steps S100 and S102 until it receives the detection result from the second recognition device 30, and continues to follow the heavy machinery.

The second recognition device 30 receives the image data (step S104). The second recognizing device 30 performs heavy machinery detection processing by processing using POLO based on the image data received in parallel with the processing of the first recognizing device 20 (step S106). The second recognition device 30 transmits the detection result based on the POLO processing to the first recognition device 20 (step S108).

After obtaining the detection result from the second recognition device 30, the first recognition device 20 compares the detection result of heavy machinery by processing using ATM3D and the detection result of heavy machinery by processing using POLO (step S110). The first recognition device 20 determines whether or not the IoU value satisfies the criterion that it is equal to or greater than a predetermined value, and adopts one of the results as the ATCS result according to the determination result (step S112). The first recognition device 20 controls the pan head of the imaging unit 10 to cause the camera to follow the heavy machine based on the adopted ATCS result (step S114).

In order to confirm the operation of the vehicle recognition system 1, an experiment was conducted in a laboratory. In order to simulate the on-site environment, we prepared a surveillance camera that would be used at an actual unmanned construction site, and installed the camera at a height of about 80 cm from the floor. In addition, as an alternative to the remote-controlled heavy machinery, we prepared two RC model dump trucks and backhoes that operate wirelessly. Also, the first recognition device 20 and the second recognition device 30 were the same as those used in the actual field experiments.

Next, the communication environment will be explained. The GPU-equipped PC is connected to the facility LAN by wire. In the preliminary experiment, the monitoring camera was connected to the facility LAN by wire, and the processing PC was connected to the facility LAN by Wi-Fi (registered trademark).

14 and 15 are diagrams showing an example of the operation result of the vehicle recognition system 1. FIG. The rectangle in the center of each screen is the center area. At t=30 in FIG. 14, the center of the heavy equipment deviates from the central area, indicating the course of movement of the pan head of the camera. As described above, according to the vehicle recognition system 1, even if the entire screen is detected as a moving object due to camera operation or zooming, the result is ignored, and heavy machinery can continue to follow the final output result. Further, according to the vehicle recognition system 1, even in a situation where the appearance of the heavy machinery changes significantly, it is possible to continue following the heavy machinery as a result of changing the template image size based on the motion detection result.

However, in the detection using POLO in the processing of the vehicle recognition system 1, there was a situation where detection was not possible in some situations due to the relationship between the plane and direction of the heavy machinery reflected in the camera, the size of the heavy machinery with respect to the screen, etc. , Based on this result, new training data was added to POLO and re-learning was performed.

Next, in order to verify the vehicle recognition system 1, an experiment was conducted using actual heavy equipment at an outdoor test site simulating the actual site. The heavy machinery used in the experiment was assumed to be used in an actual unmanned construction site, and the first recognition device 20, the second recognition device 30, and the imaging unit 10 were the same as in the preliminary experiment. The camera was installed on the roof of a two-story building so that it could overlook the entire experimental site.

Next, the communication environment will be explained. The second recognition device 30 is connected to the facility via a wired LAN as in the preliminary experiment. In order to conduct field experiments, the camera of the imaging unit 10 is directly connected to the first recognition device 20 via a wired LAN, and the first recognition device 20 is connected to the network NW via a mobile line router using Wi-Fi. bottom. There were two heavy machines of the same model within the experimental range, one of which remained stationary and the other of which was moved remotely. FIG. 15 is a photograph of the experimental environment.

In order to quantitatively evaluate the tracking ability of heavy machinery, correct data for detection was prepared by the following procedure. First, we prepared an evaluation video of the heavy machinery being tracked, which was acquired in the experiment. Next, the bounding box of the heavy machinery was set by the user's operation at regular frame intervals from this moving image. The accuracy of detection is evaluated by comparing the position and size of the bounding box obtained here and the detection result obtained by image processing. We defined two evaluation values that indicate the "following time rate" and the "detection rate" for the entire evaluation video.

A specific calculation method for the two evaluation values will be described. An IoU value indicating the degree of overlap between the correct bounding box and the rectangle obtained as the detection result is calculated for all frames having correct answer data, and the number of frames with an IoU value of 0.7 or more is counted. By dividing this by the total number of frames, the detection rate was quantified.

The higher this number, the longer the detection accuracy was maintained, but since it is possible to follow even if the IoU value is low, this number perfectly reflects the tracking ability. is not. Therefore, as another evaluation criterion, the number of frames from the start of tracking of the heavy machinery to the frame at which the tracking is no longer possible is counted and divided by the total number of frames to quantify the tracking time rate. This makes it possible to evaluate how long the heavy equipment has been chased. A frame in which tracking becomes impossible is a frame in which the IoU value of the correct data and the detection result becomes 0 for the first time.

In order to compare these two evaluation values for each method, tracking was performed using multiple methods using evaluation images. The evaluation video has a duration of 2 minutes and 39 seconds, a size of 976x552 and a frame rate of 29.96 FPS. In any tracking method, tracking is started by specifying the position or image of the bounding box to be tracked in the initial frame. In other words, all methods started tracking under the same conditions.

Next, the subject of evaluation will be described. In order to confirm the effectiveness of the proposed method, it was verified using an existing general tracking method such as MedianFlow. As general tracking methods, four methods of MedianFlow, KCF, MIL, and Boosting were implemented using the OpenCV library, and tracking was performed using each method using the evaluation video. Note that these methods require a heavy operation, so we waited until the detection result was obtained for each frame.

Further, follow-up was performed only by ATM3D and only by POLO. Since the POLO detection accuracy depends on the communication speed, two detection results were evaluated. One waits until the result of POLO is returned for each frame, performs detection in every frame, and obtains the result. On the other hand, the ATCS operating in a communication environment similar to the field environment was operated, and the detection results obtained only by the POLO before integration of the detection results were obtained. Evaluation values were obtained based on these results. In addition, the detection results of ATCS were obtained, and the evaluation values thereof were also obtained.

FIG. 16 shows detection results for each fixed frame when tracking is performed by a general tracking method using an evaluation video. f represents the number of frames of the moving image, and indicates how many frames the image has from the start of tracking. Each rectangle represents the correct data, medianflow, KCF, MIL, Boosting, and ATCS detection results.

Next, evaluation results of the first method (ATM3D) will be described. FIG. 17 shows the detection result for each fixed frame when tracking is performed only by the first method in the evaluation video. Rectangles represent detection results.

Next, evaluation results of the processing (ATCS) of the vehicle recognition system 1 will be described. FIG. 18 shows the detection result for each fixed frame when tracking is performed by the processing of the vehicle recognition system 1 using the evaluation video. Each rectangle represents the detection result of POLO and ATCS.

FIG. 19 is a diagram showing an example of evaluation results of the detection rate of heavy equipment based on each detection method.

For MedianFlow and KCF, the detection position of the heavy machinery did not change greatly from the initial position, and the heavy machinery moved as it was, so it became impossible to follow up early. MIL was able to follow for a longer time than the above two methods, but the follow-up time rate was 57%, and follow-up could not be continued.

In the follow-up by Boosting, the follow-up time rate was 100%, but in the final frame, it deviated significantly from the machine in the direction opposite to the traveling direction. Not only is there a possibility that images cannot be provided, but it is also expected that depending on how the heavy machinery moves after that, it will not be possible to follow it.

Motion detection operating in ATM3D has a low detection rate due to the nature of the algorithm, especially when the movement of the heavy machinery is fast, the size of the heavy machinery is detected to be slightly larger than the actual size. On the other hand, ATM3D has a follow-up time rate of 100%, and even in the final frame, it surrounds the entire heavy machinery.

Since POLO depends on the communication speed, we evaluated the result of ignoring the communication speed and the result of reproducing the field environment. Since POLO detects heavy machinery other than the target to follow and "(albeit with a low probability) it may misdetect places that are not heavy machinery as heavy machinery", it ignores other detection results as long as the heavy machinery to be tracked is detected. Also, when the communication speed is taken into consideration, the POLO detection result cannot be acquired in all frames. Therefore, the previous POLO detection result was fixed and evaluated until a new detection result was returned.

In POLO detection, there were no frames in which heavy machinery was not detected in all frames of the evaluation video. Also, regardless of the communication environment, it was possible to follow up to the end. This is because POLO performs detection on only one frame, unlike other techniques that rely on past frames or results for detection. When the communication environment is ignored, that is, the detection rate is very high only with pure POLO. On the other hand, in experiments that assume a real environment, a decrease in the detection rate can be seen. This is because the position of the heavy machinery in the screen changes due to the movement of the heavy machinery, especially the movement of the camera, during the time until the detection result returns, and the position of the detection result lags behind the actual position and deviates significantly.

ATCS also has a follow-up time rate of 100% and a high detection rate. On the other hand, it was confirmed that the detection position is delayed when the camera moves, similar to POLO, which considers the communication environment.

The detection rate order is the same as the tracking time rate order, and it can be confirmed that only the above-mentioned proposed method and boosting were able to follow the heavy machinery in the evaluation video. Focusing on the center position of the detection result rectangle of the final frame, it can be seen that Boosting is outside the heavy machinery, while the proposed method is near the center of the heavy machinery. It is desirable that the center position of the detection result rectangle be close to the center position of the heavy machinery. This is because the camera platform is controlled based on the center position of the detection result rectangle.

Comparing the detection rates of ATCS, POLO, and ATM3D, ATCS was the highest, and it was confirmed that tracking performance was higher than when POLO and ATM3D were used without integration. Also, it can be seen that POLO has the highest detection accuracy when the communication environment is not considered. However, when assuming a real environment, ATCS has the highest detection accuracy.

[Modification]
As mentioned above, the accuracy of ATCS depends on the communication speed between the first recognition device 20 and the second recognition device 30 . There is no problem when the high-speed communication environment is in place, but when the communication speed becomes slow, the vehicle recognition system 1 presents the position of the bounding box surrounding the heavy machinery at the time when the second recognition device 30 detects the heavy machinery. As a result, the delay increases and becomes a factor that lowers the detection rate.

In particular, when the camera platform rotates, the effect of the decrease in detection accuracy is noticeable. There is a possibility that an operation that causes the table to vibrate will occur. If a misaligned position caused by a delay in communication speed is adopted, there is a possibility that the heavy machinery will actually protrude from the detection result position, or that the heavy machinery will not appear in the detection result.

If the template image is updated based on the shifted position of the heavy equipment, there is a possibility that the template image will be generated with the background mixed with the heavy equipment, or the background image without the heavy equipment will be generated. When a template image including a background in which the heavy machinery is not accurately captured is generated, the vehicle recognition system 1 operates to follow the area including the background, and cannot perform correct tracking. Although the detection rate of ATCS is not low, there is a possibility that tracking cannot be performed due to these factors, and there is room for improvement.

ATCS+, which is a system for improving the detection rate of the above-mentioned ATCS, will be described below.

In order to improve the detection rate of ATCS, the inventors have improved the integration method of POLO and ATM3D and developed ATCS+. FIG. 20 compares the detection results of the embodiment and its modification.

At ATCS, only POLO detection positions were received and integrated into ATCS. On the other hand, in ATCS+, the frame image transmitted to the second recognition device 30 which processes POLO is stored in the first recognition device 20, and the first recognition device 20 receives the second recognition device after POLO processing. Based on the detected position information returned from 30, an image area (third image area) of the heavy machinery is cut out from the stored frame image at the time of transmission. The first recognition device 20 performs template matching using the image area cut out in the next frame as a template image.

In other words, in ATCS, the position coordinates of heavy machinery may change rapidly due to the movement of the camera. Solved the problem of template matching lag caused by

The evaluation results of ATCS+ will be described below. FIG. 21 is a diagram showing detection results of ATCS and ATCS+. FIG. 21 shows frame images of scenes in which the difference in detection accuracy was particularly large between ATCS and ATCS+, where ATCS+ is indicated by (1) and ATCS by (2). These scenes are scenes in which the camera pan head has moved greatly.

In FIG. 21, it can be confirmed that the bounding box indicating the detection result of ATCS is largely misaligned, whereas the bounding box indicating the detection result of ATCS+ stably indicates the correct position. As shown in FIG. 22, it is clear from the evaluation value calculation results that the detection results of ATCS+ are more stable than those of ATCS.

As described above, according to the vehicle recognition system 1, heavy machinery can be stably recognized based on surveillance camera images. That is, the vehicle recognition system 1 integrates image processing that combines template matching and motion detection, and two types of object detection algorithms based on machine learning. It is possible to follow an object at an unmanned construction site, which is difficult to realize by application.

In addition, according to the vehicle recognition system 1, by performing ATM3D processing, which is a proposed method that improves conventional image processing, template matching and motion detection are used to detect objects for a long time compared to other general methods. can follow. Further, according to the vehicle recognition system 1, a model obtained by extending an existing neural network model using a detection algorithm based on machine learning can more robustly detect heavy machinery.

In addition, according to the vehicle recognition system 1, by solving problems caused by network delays and camera direction changes, it is possible to more robustly follow the heavy machinery with the camera.

DESCRIPTION OF SYMBOLS 1... Vehicle recognition system 1, 10... Imaging part, 20... 1st recognition apparatus, 30... 2nd recognition apparatus, NW... Network

Claims (4)

an imaging unit that captures an image including heavy machinery;
an image processing unit that compares a template image including the heavy equipment with an image captured by the imaging unit by template matching, detects movement of the heavy equipment from the captured image, and extracts an image area including the heavy equipment; , and
The image processing unit estimates the position of the extracted image area on the image, updates the extracted image area as a new template image, and updates the updated template image with the new image captured by the imaging unit. extracting the image area containing the heavy machinery within a predetermined distance from the position in the new image when performing the next template matching based on the image ;
comparing a template image including the heavy machinery and the captured image by template matching to extract a first image region including the heavy machinery from the captured image;
individually recognizing a plurality of parts constituting the heavy machinery by applying results of deep learning by a neural network based on the image captured by the imaging unit;
recognizing the overall shape of the heavy equipment by combining the recognition results of the plurality of parts;
extracting a second image region containing the heavy equipment;
Obtaining information of the second image area, comparing the first image area and the second image area, and determining the second image area as the recognition result of the heavy machinery when the comparison result satisfies a criterion;
vehicle recognition system. The image processing unit obtains information about a first position of the second image region, and extracts a third image region including the heavy machinery in the image based on the information of the first position from the image. The third image region is updated as a new template image, and the next template matching is performed based on the updated template image and the image newly captured by the imaging unit to extract the image region including the heavy machinery. ,
A vehicle recognition system according to claim 1 . the computer
Take an image including heavy equipment,
comparing a template image including the heavy machinery with a captured image by template matching to detect movement of the heavy machinery from the captured image, extracting an image region including the heavy machinery;
estimating the position of the extracted image region on the image;
updating the extracted image area as a new template image;
extracting the image area including the heavy machinery within a range of a predetermined distance from the position in the new image when performing the next template matching based on the updated template image and the captured new image ;
comparing a template image including the heavy machinery and the captured image by template matching to extract a first image region including the heavy machinery from the captured image;
Individually recognizing a plurality of parts constituting the heavy machinery by applying results of deep learning by a neural network based on the captured image,
recognizing the overall shape of the heavy equipment by combining the recognition results of the plurality of parts;
extracting a second image region containing the heavy equipment;
Obtaining information of the second image area, comparing the first image area and the second image area, and determining the second image area as a recognition result of the heavy machinery when the comparison result satisfies a criterion;
Vehicle recognition method. Obtaining information about a first position of the second image region, extracting a third image region including the heavy machinery in the image based on the information of the first position from the image;
updating the extracted third image region as a new template image;
performing the following template matching based on the updated template image and the newly captured image, and extracting the image region including the heavy equipment;
The vehicle recognition method according to claim 3 .

Images