JP7802275B2 - Monitoring and warning device, monitoring and warning method, and program - Google Patents

Description

The present invention relates to a monitoring and warning device that, when a vehicle enters a predetermined monitoring area, checks the danger of the vehicle and outputs a warning if the danger exists.
For example, this relates to a device that detects vehicles approaching from behind, determines the risk, and issues a warning in the event of danger, in order to ensure the safety of inspection workers who temporarily park an inspection vehicle on the shoulder or leftmost lane of a highway and get out of the vehicle to visually inspect the surrounding conditions.
The present invention also relates to a method executed by the above device and a program for causing a computer to function as the above device.

Japanese Patent Laid-Open Publication No. 2017-163374 (Patent Document 1) discloses a technology for accurately estimating the behavior of an object moving on a road within a predetermined area on the road, based on an image input from an imaging device that captures the area.
Japanese Patent Laid-Open Publication No. 2018-022234 (Patent Document 2) discloses a technology for accurately detecting objects such as other vehicles present around a vehicle from captured images.

Object detection using deep learning is called object detection, and has been published in overseas papers (Non-Patent Documents 1 and 2) since around 2016. It has also been implemented in open source software, and by spring 2018, real-time processing (30 frames per second or more) was possible using existing computers.
For example, in object detection and identification (classification) using deep learning, relatively well-known implementations such as SSD and YOLO use transfer learning, which uses a trained model that uses a huge number of training images, making it possible to build a model using original data with a minimal amount of computation.

One example of an object tracking technique is Deep-SORT, an extension of SORT (Simple Online and Realtime Tracking). Deep-SORT is described, for example, in Non-Patent Document 3.

Japanese Patent Application Laid-Open No. 2017-163374 JP 2018-022234 A YOLOv3: An Incremental Improvement / Joseph Redmon, Ali Farhadi / University of Washington SSD: Single Shot MultiBox Detector / Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg https:nanonets.com/blog/object-tracking-deepsort/ (as of June 16, 2020)

For example, in the case of minor inspections of expressways that do not involve main lane restrictions, an inspection vehicle is temporarily parked on the shoulder of the road and a visual inspection is carried out by walking ahead of the vehicle (in the direction of travel). However, on rare occasions, vehicles may travel on the shoulder of the road, so monitoring is required to ensure safety.
In other words, there is a demand for inspectors to ensure safety and security without being distracted by what is behind them (in the opposite direction of the vehicle's travel).
There is also a demand for an easy way to ensure safety and security.
The present invention aims to meet such demands.

Examples of the present invention are described below in [1] to [8]. Note that in this section ([Means for Solving the Problems]) and the next section ([Effects of the Invention]), the symbols are used to facilitate understanding, and are not intended to limit the present invention to the symbol configuration.
[1] Invention 1
A monitoring and warning device that monitors the rear side of a road from a shoulder 91 or the leftmost lane of a road and warns a user of a vehicle entering the shoulder 91 or the leftmost lane,
a camera 30 that captures an image of the rear side of the road from a predetermined position in the road shoulder 91 or the leftmost lane;
an image processing means 51 for detecting, identifying, and tracking a vehicle and the class to which the vehicle belongs from each image frame of the video input from the camera 30;
a boundary line setting means S21 for setting a virtual boundary line 9 in each image frame so as to overlap a white line 90 on the road surface that separates a vehicle travel lane 92 of the road from the shoulder 91 or the leftmost lane;
a threshold value storage means 61 for storing, for each vehicle class, a warning threshold value for the area ratio of two areas B1, B2 obtained by dividing a vehicle area BB occupied by a vehicle being tracked by the image processing means 51 with the boundary line 9;
area ratio processing means S21, S23 for calculating the area ratio for each vehicle being tracked by the image processing means 51 and comparing the area ratio with a warning threshold value for the class to which the vehicle belongs;
a warning means S41 for outputting a warning when the area ratio processing means S23, S31 determines that the area ratio of any vehicle being tracked by the image processing means 51 exceeds a warning threshold value for the class of the vehicle;
A monitoring and warning device comprising:

<Terminology details>
The image processing means (detector, classifier, tracker) 51 can be configured by implementing, for example, Deep-SORT and YOLOv3. Depending on the required performance, object detection and identification (classification) can be performed using, for example, SSD instead of YOLOv3. Taking into consideration the behavior of the object to be detected in the detection area and the accuracy and performance required of the object information processing device, an appropriate open source (library/platform/AI application) can be selected. Object detection and identification (classification) can be performed simultaneously, or in a two-stage application. Furthermore, it is preferable to appropriately use a large number of images of various vehicles (trucks, passenger cars, light trucks, buses, motorcycles, etc.) taken from the front as training data.
When the area ratio is defined as B1/B2 or B1/(B1+B2), "exceeding the warning threshold" means "the calculated value of the area ratio is equal to or greater than the warning threshold."
Conversely, if the area ratio is defined as B2/B1 or B2/(B1+B2), "exceeding the warning threshold" means "the calculated value of the area ratio is equal to or greater than the warning threshold."
Here, B1 refers to the area of the area on the shoulder side of the vehicle area, and B2 refers to the area of the area on the driving lane side of the vehicle area (see FIGS. 7(b) and 7(c)).
The warning threshold stored in the threshold storage means is assumed to be a threshold that conforms to the above definition. For example, if B1/B2 is used, a threshold that is determined to be dangerous for each road type based on the results of measurement by B1/B2 is assumed to be prepared for each class (truck, passenger car, light truck, bus, motorcycle, etc.). For example, in the case of a highway, since approaching vehicles are traveling at high speed, the threshold is set to a value that indicates a relatively small amount of intrusion into the shoulder area so that the inspection worker has sufficient time to take shelter.
The warning means S41 is, for example, a means for controlling a speaker to output a warning sound, or a means for sending warning data to a wireless receiver carried by an inspector.

[2] Invention 2
In Invention 1,
The vehicle area occupied by the vehicle being tracked by the image processing means 51 is the area of the bounding box BB of the vehicle.
A monitoring and warning device characterized by:
[3] Invention 3
In Invention 1 or Invention 2,
The boundary line setting means S21 sets the boundary line 9 in response to an input from the operation input device 20.
A monitoring and warning device characterized by:
[4] Invention 4
In Invention 1 or Invention 2,
When the white line 90 on the road surface is curved, the boundary line setting means S21 divides a predetermined monitoring area of the road shoulder 91 or the leftmost lane into a plurality of polygons 9a to 9e, and sets the sides of the polygons that overlap the white line 90 as approximately parts of the boundary line 9.
A monitoring and warning device characterized by:
[5] Invention 5
In the invention 4,
The polygons 9a to 9e are trapezoids formed by a plurality of parallel lines set in order at intervals from the back side to the front side within the predetermined monitoring area , with the upper and lower sides being in turn parallel lines.
A monitoring and warning device characterized by:
The terms "far side" and "near side" refer to the rear side of the road (opposite the direction of travel of the vehicle).
[6] Invention 6
In any one of Inventions 1 to 5,
The image processing means 51 is adjusted by deep learning using a large number of image data created based on photographed images of the front side of each vehicle type, such as trucks, buses, passenger cars, light trucks, and motorcycles, as training data for vehicle detection and identification.
A monitoring and warning device characterized by:
[7] Invention 7
A program for causing a computer to function as any one of the monitoring and warning devices of Inventions 1 to 6.

[8]
A monitoring and warning method for monitoring a rear side of a road from a shoulder 91 or a leftmost lane of a road and warning a user of a vehicle entering the shoulder 91 or the leftmost lane,
Detecting, identifying and tracking a vehicle and the class to which the vehicle belongs from each image frame of a video input from a camera 30 that captures an image of the rear side of the road from the road shoulder 91 or a predetermined position in the leftmost lane,
A virtual boundary line 9 is set in each image frame so as to overlap a white line 90 on the road surface that separates a vehicle driving lane 92 of the road from the shoulder 91 or the leftmost lane (S21).
For each vehicle being tracked, the area ratio of two areas B1, B2 formed by dividing the vehicle area BB occupied by the vehicle by the boundary line 9 is calculated (S21), and each is compared with the warning threshold of the class to which the vehicle belongs (S23).
When the calculated area ratio of any vehicle being tracked exceeds the warning threshold value of the class of the vehicle (S31), a warning is output (S41).
A monitoring and warning method comprising:

Invention 1 is a monitoring and warning device that monitors the rear side of a road from a shoulder 91 or the leftmost lane of a road and warns a user of a vehicle entering the shoulder 91 or the leftmost lane, the monitoring and warning device comprising: a camera 30 that captures an image of the rear side of the road from a predetermined position in the shoulder 91 or the leftmost lane; image processing means 51 that detects, identifies, and tracks a vehicle and the class to which the vehicle belongs from each image frame of a video input from the camera 30; boundary line setting means S21 that sets a virtual boundary line 9 in each image frame so as to be superimposed on a white line 90 on the road surface that separates a vehicle driving lane 92 of the road from the shoulder 91 or the leftmost lane; and threshold values that are stored for each vehicle class as warning thresholds for the area ratio of two areas B1 and B2 formed by dividing a vehicle area BB occupied by a vehicle being tracked by the boundary line 9. Since the monitoring and warning device is characterized by having a memory means 61, area ratio processing means S21, S23 in which the image processing means 51 calculates the area ratio for each vehicle being tracked and compares it with the warning threshold for the class to which the vehicle belongs, and a warning means S41 that outputs a warning when the area ratio processing means S23, S31 determines that the area ratio of any of the vehicles being tracked by the image processing means 51 has exceeded the warning threshold for the class of that vehicle, for example, during a patrol inspection of a highway that does not involve main lane restrictions, an inspection vehicle can be temporarily parked on the shoulder of the road, and when walking ahead of it (in the direction of vehicle travel) to perform a visual inspection, the inspection worker (user) can be warned at an appropriate time (a time that allows sufficient time to evacuate) that a vehicle is entering the shoulder and approaching.
For inventions 2 to 6, specific configuration examples that can achieve the above can be given.
Invention 7 can realize the above by a computer.
Invention 8 is the same as Invention 1.

1 is a block diagram illustrating the function and configuration of a monitoring/warning device according to an embodiment; 4 is a flowchart illustrating an image processing procedure of the monitoring/alarm device of FIG. 1 . FIG. 2 is a block diagram illustrating a specific configuration example of the monitoring/alarming device and peripheral devices of FIG. 1 . 4 is a flowchart illustrating an example of a processing procedure in the monitoring/alarm device of FIG. 1 . An explanatory diagram of the relationship between roads, shoulders, boundary lines, etc. (a) is an explanatory diagram of sufficient time to ensure safety (warning timing), (b) is an explanatory diagram of the vehicle area BB of a truck, and (c) is an explanatory diagram of the vehicle area of a passenger car. 1A is an explanatory diagram of a road, a shoulder, a boundary line, and a vehicle area BB; FIG. 1B is an explanatory diagram of a boundary line dividing the vehicle area; and FIG. 1C is an explanatory diagram of two areas formed by dividing the vehicle area.

An embodiment of the present invention will be described with reference to the drawings.
As shown in FIG. 1 , the monitoring/warning device of this embodiment includes a camera 30, an image processing device 5 that receives and processes images from the camera 30, an operation input device 20 that may be configured as a keyboard or game pad, a storage device 60, a wireless transmitter 58, and a control device (computer) 50 that controls each of these components. Each of these devices may be an independent device or an integrated device. Furthermore, the image processing device 5 and the control device 50 are generally implemented as an image processing board and a control board. Furthermore, the storage device 60 may be configured as a memory, a hard disk, an SSD, or a combination thereof.

The camera 30 is a device that is temporarily installed on the road shoulder 91 (see FIG. 7; in a location without a road shoulder, the leftmost lane, etc.) and is capable of capturing images of the area behind the road, which is the subject of the image capture, from a fixed position, and outputs, for example, 30 fps video images to the image processing device 5. An example of the fixed position is a position that provides a panoramic view of a predetermined range behind the road (a range in which a vehicle can be detected and tracked with sufficient timing to ensure safety; including at least the range of the monitoring area in FIG. 6(a)), as shown in each diagram in FIG. 5.
The camera 30 is temporarily fixed in this position to capture the desired moving image. The camera 30 may be configured to be operable while still mounted on the inspection vehicle. Because it is temporarily fixed, it can be easily installed and removed.

The image processing device 5 has an image processing unit 51 that detects, identifies, and tracks vehicles (trucks, passenger cars, light trucks, motorcycles, etc.) from each image frame of the video input from the camera 30. Although not shown, a display device (not shown) may be provided downstream of the image processing device 5 that can display the video input from the camera 30 as an image in which a bounding box BB is superimposed on the detected vehicle in real time, in a still state, or repeatedly, in response to instructions from the control device 50.

The image processing unit 51 can use a known AI application capable of performing image processing (detecting, identifying, and tracking each vehicle) with the performance and accuracy required depending on the characteristics of the imaged subject (such as the size and speed of the vehicle being detected). In this embodiment, a configuration based on YOLO (You Only Look Once) is used, but other known AI applications may also be used as appropriate. For example, instead of YOLO (You Only Look Once), it may be considered to use SSD (Single Shot Multibox Detector) or Faster-R-CNN, as long as the required performance is met. Alternatively, it may be considered to use Deep-SORT-YOLO.v3, etc.

The control device 50 controls each device in response to user input from the operation input device (game pad, keyboard, etc.) 20. For example, when a "start imaging" command is received from the operation input device 20, the control device 50 commands the camera 30 to begin imaging, causing it to begin imaging. The camera 30 outputs the moving images being captured, i.e., the chronological image frames, in order to the image processing unit 51 of the image processing device 5. In this embodiment, the moving images output by the camera 30 are 30 fps.

The image processing unit 51 detects and identifies (classifies, divides into categories) vehicles (passenger cars, trucks, motorcycles, light trucks, etc.) from each image frame of the video input from the camera 30, tracks them, and stores data for each vehicle (position and size of the bounding box BB, vehicle type (class), ID, image frame number, etc.) in an appropriate memory area (not shown) and outputs the data to the control device 50.

When the control device 50 receives data for the current image frame from the image processing unit 51, it executes the processing shown in Figure 2. That is, it calculates the area ratio of each of the two areas formed by dividing the vehicle area (area of the bounding box BB) of each vehicle in the current image frame by the boundary line 9, compares the calculated area ratio for each vehicle with the warning threshold for the class to which the vehicle belongs (passenger car/truck/motorcycle/light truck, etc.), and outputs a warning to the user if the area ratio exceeds the warning threshold.

The explanation will be given with reference to FIG.
The control device 50 acquires various data detected in the current image frame from the image processing device 5 (S11). For example, the control device 50 acquires the information on each vehicle detected in the current image frame, its class, and the size and position of the vehicle area (bounding box) BB of that vehicle, as well as the information on each vehicle detected in the previous image frame and being tracked in the current image frame, its class, and the size and position of the vehicle area (bounding box) BB of that vehicle.
It is assumed that data on the boundary line 9 indicating the boundary between the travel lane 92 and the road shoulder 91 has already been acquired by initial setting when the monitoring/warning device is installed, for example.

Next, using the acquired data, the area ratio of each of the two regions formed by dividing the vehicle region of each vehicle by the boundary line 9 is calculated (S21).
In this embodiment, the area ratio is calculated as B1/(B1+B2).
Here, B1 is the area of the area on the shoulder 91 side when the vehicle area (bounding box) BB is divided by the boundary line 9, and B2 is the area of the area on the driving lane 92 side (FIG. 7).
That is, in this embodiment, the area ratio is the ratio of the area on the shoulder 91 side when the vehicle area (bounding box) BB is divided by the boundary line 9 to the area of the vehicle area.

This area ratio is compared with the warning threshold for the class to which the vehicle belongs (passenger car/truck/motorcycle/light truck, etc.) (S23). This warning threshold is a value found by trial and error from the perspective of when the above-mentioned area ratio reaches a certain level, in other words, at what distance and to what extent the vehicle being detected and tracked must enter the shoulder of the road before a warning is issued to the user, so that sufficient time can be secured for the user to take shelter.
Naturally, as for the warning threshold value, values calculated according to the principle corresponding to the above definition "B1/(B1+B2)" are prepared for each class.
In this embodiment, the distance from the monitoring/warning device to the vehicle is obtained using input data from the distance sensor 70 .

If the comparison result in step S23 indicates that the area ratio of any vehicle exceeds the threshold value for the class to which the vehicle belongs (YES in S31), a warning is output (S41).
In this embodiment, "data indicating that a warning should be sent" is output to the wireless transmitter 58. As a result, the wireless transmitter 58 wirelessly transmits the data to each wireless receiver 80 carried by each inspection worker. Each wireless receiver 80 outputs a signal that will reliably attract the attention of the inspection worker. For example, a loud warning sound is output.

In Figure 1, the GNSS receiver 10 is a device that outputs the current time and current position based on information received from GNSS satellites. This allows the monitoring and warning device of this embodiment to obtain accurate position and time information for issuing the above-mentioned monitoring and warning.

In the above embodiment, "B1/(B1+B2)" is used as the area ratio. However, instead of this,
(1) “B1/B2”
(2) “B2/(B1+B2”)
(3) “B2/B1”
In that case, the threshold for each class should naturally be a threshold that conforms to the definition.
In the above cases (2) and (3), the numerical value of the area ratio decreases as the vehicle advances further toward the road shoulder 91. Therefore, "exceeding the threshold" means "going beyond the threshold and the area ratio decreasing."

The above embodiment describes a case where the monitoring and warning device of the present invention is installed on the shoulder of a highway, but the present invention is not limited to such an embodiment. For example, when installing the device at the entrance or exit of a service area, similar processing is possible by setting an appropriate area as the danger area to be monitored instead of the shoulder. Alternatively, the present invention can be applied to places such as train station platforms where trains cannot be seen early because the approach tracks are curved, by setting a danger area to be monitored in the same way instead of the shoulder.

10 GNSS receiver 20 Operation input device 30 Camera 40 Power supply device 5 Image processing device 51 Image processing unit 50 Control device 58 Wireless transmitter 60 Storage device 61 Threshold memory unit 70 Distance sensor 80 Wireless receiver 9 Boundary line 9a, 9b, 9c, 9d, 9e Trapezoid 90 White line strip 91 Shoulder or leftmost lane 92 Vehicle driving lane BB Vehicle area B1 Area on the shoulder side within the vehicle area B2 Area on the driving lane side within the vehicle area

Claims (10)

A monitoring and warning device that monitors the rear side of a road from a shoulder or the leftmost lane of a road and warns a user of a vehicle entering the shoulder or the leftmost lane,
a camera that captures an image of the rear side of the road from a predetermined position in the road shoulder or the leftmost lane;
an image processing means for detecting, identifying, and tracking a vehicle and the class to which the vehicle belongs from each image frame of the video input from the camera;
a boundary setting means for setting a virtual boundary line within each of the image frames so as to overlap a white line on a road surface that separates a vehicle driving lane of the road from the shoulder or the leftmost lane;
a threshold storage means for storing, for each vehicle class, a warning threshold value for the area ratio of two areas obtained by dividing a vehicle area occupied by a vehicle being tracked by the image processing means by the boundary line;
an area ratio processing means for calculating the area ratio for each vehicle being tracked by the image processing means and comparing the area ratio with a warning threshold value for the class to which the vehicle belongs;
a warning means for outputting a warning when the area ratio processing means determines that the area ratio of any vehicle being tracked by the image processing means exceeds a warning threshold value for the class of the vehicle;
A monitoring and warning device comprising: In claim 1,
The vehicle area occupied by the vehicle being tracked by the image processing means is the area of the bounding box of the vehicle.
A monitoring and warning device characterized by: In claim 1 or claim 2,
the boundary line setting means sets the boundary line in response to an input from an operation input device.
A monitoring and warning device characterized by: In claim 1 or claim 2,
When the white line on the road surface is curved, the boundary line setting means divides the predetermined monitoring area of the road shoulder or the leftmost lane into a plurality of polygons, and sets the sides of the polygons that overlap the white line as approximately parts of the boundary lines.
A monitoring and warning device characterized by: In claim 4,
The polygon is a trapezoid formed by using a plurality of parallel lines, which are set in order at intervals from the back side to the front side within the predetermined monitoring area, as upper and lower sides in turn.
A monitoring and warning device characterized by: A monitoring and warning device that monitors from within a predetermined monitoring area adjacent to a vehicle's driving area and warns a user of a vehicle entering the predetermined monitoring area ,
a camera that captures images from a predetermined position within the predetermined monitoring area ;
an image processing means for detecting, identifying, and tracking a vehicle and the class to which the vehicle belongs from each image frame of the video input from the camera;
a boundary setting means for setting a virtual boundary line within each of the image frames so as to overlap a linear band that separates the predetermined monitoring area from a vehicle travel area adjacent to the monitoring area ;
a threshold storage means for storing, for each vehicle class, a warning threshold value for the area ratio of two areas obtained by dividing a vehicle area occupied by a vehicle being tracked by the image processing means by the boundary line;
an area ratio processing means for calculating the area ratio for a vehicle being tracked by the image processing means and comparing the area ratio with a warning threshold for the class to which the vehicle belongs;
a warning means for outputting a warning when the area ratio processing means determines that the area ratio of the vehicle being tracked by the image processing means exceeds a warning threshold value for the class of the vehicle;
A monitoring and warning device comprising: In claim 6,
When the linear band is curved, the boundary line setting means divides the predetermined monitoring area into a plurality of polygons and sets sides of the polygons that overlap the linear band as approximately parts of the boundary lines .
A monitoring and warning device characterized by: In any one of claims 1 to 7 ,
The image processing means is adjusted by deep learning using a large number of image data created based on photographed images of the front side of each vehicle type, such as trucks, buses, passenger cars, light trucks, and motorcycles, as training data for vehicle detection and identification.
A monitoring and warning device characterized by: A program for causing a computer to function as the monitoring/warning device according to any one of claims 1 to 8 . A monitoring and warning method for monitoring a rear side of a road from a shoulder or a leftmost lane of a road and warning a user of a vehicle entering the shoulder or the leftmost lane, comprising:
Detecting, identifying and tracking a vehicle and the class to which the vehicle belongs from each image frame of a video input from a camera that captures an image of the rear side of the road from a predetermined position in the shoulder or the leftmost lane of the road;
a virtual boundary line is set in each of the image frames so as to overlap a white line on the road surface that separates the vehicle driving lane of the road from the shoulder or the leftmost lane;
For each vehicle being tracked, calculate the area ratio of two regions obtained by dividing the vehicle region occupied by the vehicle by the boundary line, and compare the area ratio with the warning threshold of the class to which the vehicle belongs;
outputting a warning when the calculated area ratio of any of the vehicles being tracked exceeds a warning threshold for the class of the vehicle;
A monitoring and warning method comprising: