JP6998726B2 - Monitoring equipment, monitoring system, monitoring method and monitoring program - Google Patents

Description

The present invention relates to a monitoring device, a monitoring system, a monitoring method and a monitoring program.

Conventionally, a technique of mounting a camera on an automobile to monitor the outside has been proposed (for example, Patent Document 1).

Japanese Patent No. 4161225

By the way, the above-mentioned technique is effective when the driver is in a car. However, it is not possible to prevent the vehicle from being harmed when the driver is away from the vehicle while the vehicle is parked.

The present invention attempts to solve such a problem, and is a monitoring device, a monitoring system, a monitoring method, and a monitoring device capable of giving a notification according to a situation outside the vehicle when the driver is away from the vehicle. The purpose is to provide a monitoring program.

The first invention is a monitoring device for monitoring the outside of an automobile, which is an image data acquisition means for acquiring image data of an object and an object recognition means for recognizing a monitoring target among the objects included in the image data. When the object recognition means recognizes the monitoring target, the notification condition determination means for determining whether or not the notification condition for notifying the outside is satisfied, and the external notification mode for satisfying the notification condition. It has a notification means for notifying the user and a speed change recognition means for recognizing a change in the speed at which the monitoring object approaches the vehicle, and the notification condition is a monitoring device for increasing the speed . be.

According to the configuration of the first invention, the monitoring device can acquire image data by the image data acquisition means, and can recognize the monitoring target among the objects included in the image data by the object recognition means. Then, it is possible to determine whether or not the notification condition is satisfied by the notification condition determination means, and notify the outside by the notification means. This makes it possible to give a notification according to the situation outside the vehicle when the driver is away from the vehicle.

The second invention, in the configuration of the first invention, is an unmanned monitoring device that refers to data generated by deep learning.

The third invention has a distance estimation means for estimating the distance between the monitoring target and the automobile in the configuration of either the first invention or the second invention, and the notification condition is the monitoring. A monitoring device in which the object is located within a predetermined distance from the automobile.

A fourth aspect of the invention is a monitoring device having a notification mode determining means for determining a notification mode to be notified to the outside based on the distance between the monitoring target and the automobile in the configuration of the third invention.

The fifth invention has an approach determination means for determining whether or not the monitoring target is approaching the automobile in the configuration of the first invention or the second invention, and the notification condition is the monitoring target. Is a monitoring device that is close to the automobile.

The seventh invention is the position information indicating the position of the automobile in the configuration of the first invention or the second invention, when the notification means notifies the outside that the notification condition is satisfied by a predetermined notification mode. Is a monitoring device having a tracking means for tracking the vehicle.

The eighth invention is a monitoring system for monitoring the outside of an automobile, in which an image data acquisition means for acquiring image data of an object photographed by the photographing means and a monitoring target among the objects included in the image data are monitored. It is determined that the object recognition means to be recognized, the notification condition determination means for determining whether or not the notification condition for notifying the outside is satisfied when the object recognition means recognizes the monitoring target, and the notification condition are satisfied. It has a notification means for notifying the outside according to the notification mode of the above, and a speed change recognition means for recognizing a change in the speed at which the monitoring target approaches the vehicle, and the notification condition is that the speed has increased. A monitoring system.

A ninth aspect of the invention is an image data acquisition step in which a monitoring device for monitoring the outside of an automobile acquires image data of an object, an object recognition step for recognizing a monitoring target among the objects included in the image data, and the like. When the monitoring target is recognized in the object recognition step, the notification condition determination step for determining whether or not the notification condition for notifying the outside is satisfied and the notification that the notification condition is satisfied are notified to the outside by a predetermined notification mode. The notification step includes a speed change recognition step for recognizing a change in the speed at which the monitoring object approaches the vehicle, and the notification condition is a monitoring method in which the speed has increased .

The tenth invention is a control device for managing a monitoring device for monitoring the outside of an automobile, an image data acquisition means for acquiring image data of an object, and an object for recognizing a monitoring target among the objects included in the image data. A recognition means, a notification condition determination means for determining whether or not a notification condition for notifying the outside is satisfied when the object recognition means recognizes the monitoring target, and a predetermined notification mode for satisfying the notification condition. It functions as a notification means for notifying the outside by means of a speed change recognition means for recognizing a change in the speed at which the monitoring object approaches the vehicle, and the notification condition is a monitoring program in which the speed has increased. be.

As described above, according to the present invention, when the driver is away from the automobile, it is possible to give a notification according to the situation outside the automobile.

It is a figure which shows the outline of the Embodiment of this invention. It is a schematic diagram which shows the functional block of a photographing apparatus. It is a schematic diagram which shows the functional block of a monitoring device. It is a conceptual diagram for explaining the feature data for image recognition. It is a figure for demonstrating the notification condition. It is a figure for demonstrating the notification condition. It is a conceptual diagram which shows the notification mode data. It is a figure for demonstrating a tracking program. It is a flowchart which shows the operation of a monitoring device and the like.

Embodiments of the present invention will be described with reference to the drawings. The configuration that can be appropriately implemented by those skilled in the art will be omitted, and only the basic configuration of the present invention will be described.

As shown in FIG. 1, the monitoring device 100 is housed in the court pocket of the driver 350 away from the automobile 300. The monitoring device 100 is, for example, a mobile phone (including a smartphone). The driver 350 parks the car 300 in the parking space on the shoulder of the road R1 and goes out for a meal with the lady 352. A photographing device 10 is arranged near the inner upper portion of the windshield of the automobile 300. A camera (not shown) constituting the photographing apparatus 10 is an example of photographing means. The photographing device 10 is, for example, a drive recorder.

The monitoring device 100 acquires (receives) image data of an image taken by the photographing device 10 from the photographing device 10 when the driver 350 is away from the automobile 300.

The monitoring device 100 recognizes the monitoring target when the monitoring target is included in the image data received from the photographing device 10 by image recognition. The monitoring target is, for example, a human being in a predetermined dress, and humans 360 and 362 are applicable. Humans 360 and 362 wear red jackets 360a and 362a, respectively. The monitoring device 100 recognizes the monitoring target in the image data, and when the notification condition is satisfied, the monitoring device 100 notifies the driver 350 by sound or vibration that the notification condition is satisfied. Further, the monitoring device 100 requests the photographing device 10 for position information indicating the position of the photographing device 10 when a predetermined condition is satisfied. The position of the photographing device 10 is also the position of the automobile 300. As a result, when the automobile 300 is moved to a person other than the driver 350, the monitoring device 100 can track the automobile 300 by receiving the position information from the photographing device 10.

As shown in FIG. 2, the photographing apparatus 10 includes a CPU (Central Processing Unit) 50, a storage unit 52, a wireless communication unit 54, a GPS (Global Positioning System) unit 56, an image processing unit 60, and a power supply unit 70. ..

The photographing device 10 can acquire an external image by operating a camera (not shown) which is a central configuration of the photographing device 10 by the image processing unit 60.

The photographing device 10 can transmit image data to the monitoring device 100 by the wireless communication unit 54.

The photographing device 10 can measure the position of the photographing device 10 itself by the GPS unit 56. The GPS unit 56 basically receives radio waves from four or more GPS satellites (navigation satellites) and measures the position of the photographing device 10.

The power supply unit 70 is connected to the power source of the automobile 300 and is supplied with electric power from the automobile 300.

In the storage unit 52, in addition to a general program as a drive recorder, a stop shooting program, an image data transmission program, and a position information transmission program are stored. The CPU 50 and the stop shooting program are examples of the stop shooting means. The CPU 50 and the image data transmission program are examples of image transmission means. The CPU 50 and the position information transmission program are examples of the position information transmission means.

The photographing device 10 photographs the outside while the automobile 300 is stopped by the stopping photographing program. The photographing device 10 continuously photographs the outside while the automobile 300 is running, but even when the automobile 300 is stopped, the driver 350, who is the user, switches the camera of the photographing device 10. Unless the shooting is interrupted, such as when the camera is turned off, external shooting is continued.

The photographing device 10 is adapted to continuously transmit the image data of the image captured while the vehicle is stopped to the monitoring device 100 by the image data transmission program.

The photographing device 10 continuously transmits the position of the photographing device 10 to the monitoring device 100 when a predetermined condition is satisfied by the position information transmission program. The predetermined condition is that the request for transmission of the position information is received from the monitoring device 100.

As shown in FIG. 3, the monitoring device 100 includes a CPU 150, a storage unit 152, a wireless communication unit 154, a GPS unit 156, and a power supply unit 170.

The monitoring device 100 can receive image data from the photographing device 10 by the wireless communication unit 154. Further, it is possible to transmit a position information transmission request to the photographing apparatus 10 and receive the position information.

The monitoring device 100 can measure the position of the monitoring device 100 itself by the GPS unit 156.

The power supply unit 170 is a rechargeable battery that can be charged, and supplies electric power to each unit of the monitoring device 100.

In the storage unit 152, in addition to a general program as a mobile phone, an image data acquisition program, an object recognition program, a distance estimation program, an approach judgment program, a speed change recognition program, a notification condition judgment program, a notification mode determination program, and a notification The program and the tracking program are stored. The CPU 150 and the image data acquisition program are examples of image data acquisition means. The CPU 150 and the object recognition program are examples of the object recognition means. The CPU 150 and the distance estimation program are examples of distance estimation means. The CPU 150 and the approach determination program are examples of the approach determination means. The CPU 150 and the speed change recognition program are examples of speed change recognition means. The CPU 150 and the notification condition determination program are examples of the notification condition determination means. The CPU 150 and the notification mode determination program are examples of the notification mode determination means. The CPU 150 and the notification program are examples of notification means. The CPU 150 and the tracking program are examples of tracking means.

The monitoring device 100 acquires image data from the photographing device 10 by an image data acquisition program.

The monitoring device 100 recognizes an object included in the image data by the object recognition program, and recognizes the monitoring target. In the present embodiment, it is assumed that the monitoring target is a human being wearing a red jacket.

The monitoring device 100 refers to the feature data generated by deep learning by the object recognition program. Here, deep learning is machine learning of a multi-layered neural network, and the field of image recognition is one of the promising fields of utilization.

The monitoring device 100 identifies the features of the object included in the image data acquired from the photographing device 10 based on a large number of image recognition feature data for each category (hereinafter, also referred to as "feature information"), and the object. You can recognize the category of. The feature information may be stored in the storage unit 152 or stored in an external storage device so that the monitoring device 100 can access the external storage device. Recognition in this category is called general object recognition. As shown in the conceptual diagram of the feature data for image recognition in FIG. 4, the categories are, for example, "human", "dog", "cat" and the like. Each category is subdivided, and there are subcategories 1 such as "male" and "female" in the "human" category, and "jacket" and "jacket" in the "male" of subcategory 1. There is a sub-category 2 such as "pants", and there is a sub-category 3 such as "red" and "blue" in the "jacket" of the sub-category 2. The storage unit 152 stores a large number of feature data for each of the above-mentioned categories. The monitoring device 100 can refer to, for example, the characteristic information of the "human" acquired based on a large number of image examples of the "human".

In general object recognition, for example, a large number of features such as contours and directions of individual configurations are extracted from image data, and the correlation (degree of correlation) is determined by comparing them with the features of each category acquired by deep learning. The higher the degree of correlation, the more likely the object in the acquired image belongs to a particular category. For example, when the correlation degree is 0, the possibility of belonging to a specific category (hereinafter referred to as "category common probability") is 0, and when the correlation degree shows the maximum value, the category common probability is 100%. Is defined as. The monitoring device 100 determines that the category of the object in the acquired image belongs to a specific category when the category common probability is a predetermined reference value, for example, 95% or more.

When the acquired image includes a male human wearing a red jacket, the monitoring device 100 refers to the feature data by the above-mentioned general object recognition, and is a "human", a "male", and the like. Wearing a "jacket", the jacket is recognized as "red", and the object in the image is recognized as a monitoring target.

The monitoring device 100 estimates the distance between the monitored object and the automobile 300 by the distance estimation program. The number of pixels of the photographing apparatus 10 is, for example, 3 million pixels. The number of pixels of the photographing device 10 and the number of pixels of the monitoring device 100 do not necessarily have to be the same, but in the present embodiment, the total number of images of the monitoring device 100 is the same as the total number of images of the photographing device 10.

As shown in FIG. 5A, the monitoring target occupies a predetermined number of pixels in the image data according to its size. The larger the number of pixels occupied by the monitored object, the closer the distance to the automobile 300. The monitoring device 100 has, for example, the number of pixels in the height direction when a rod having a height of 1 m (meter) is 500 m away, the number of pixels when it is 200 meters away, and the position 100 meters away. It has basic reference values such as the number of pixels at a certain time, the number of pixels at a position 50 mails away, and the number of pixels at a position 10 meters away. Further, for each category, for example, in the case of "human" of "male", the average height is 170 cm (centimeter), which is a category reference value. The monitoring device 100 is adapted to estimate the distance between the monitoring target and the automobile 300 based on the number of pixels occupied by the actual monitoring target with reference to the basic reference value and the category reference value. For example, in the case of a "human" of a "male", the average height is 170 cm, so the number of pixels in the height direction at a predetermined distance is 1.7 times the basic reference value A based on a height of 1 m. Calculate as a number (A × 1.7). In this embodiment, the estimated distance does not need to be accurate, and a simple configuration is desirable. Therefore, the distance is estimated based on the number of pixels in the image.

The monitoring device 100 determines whether or not the monitored object is approaching the automobile 300 by the approach determination program. For example, as shown in FIG. 6, the number of pixels occupied by the monitored object at time t2, which is the time Δt elapsed from time t1, is larger than the number of pixels occupied by the monitored object at time t1 (see FIG. 6A). b) When there are many), the monitoring device 100 determines that the monitoring target is close to the automobile 300.

The monitoring device 100 recognizes a change in speed at which the monitored object approaches the automobile 300 by the speed change recognition program. In the monitoring device 100, for example, the increase δB1 in the number of pixels occupied by the monitored object between the time t1 and the time t2 when the time Δt has elapsed is the time t3 when the time t2 and the time Δt have elapsed, which is the subsequent time zone. If the increase in the number of pixels occupied by the monitoring target is smaller than the increase δB2 (δB1 <δB2), it is determined that the monitoring target is increasing the speed.

The monitoring device 100 determines whether or not to notify the driver 350, who is the user of the monitoring device 100, of the establishment of the notification condition by the notification condition determination program. As for the notification condition, for example, as shown in FIG. 5A, when one monitoring target is recognized in the image data, as shown in FIG. 5B, two monitoring targets are included in the image data. When it is recognized, or when the monitored object is approaching the vehicle 300 as shown in FIGS. 6 (a) and 6 (b), or when the monitored object is approaching the vehicle 300 at an increased speed. be. The notification condition can be selected and set in advance by the driver 350 who is the user of the monitoring device 100.

The monitoring device 100 determines the notification mode to be notified to the outside (driver 350) based on the distance between the monitoring target and the automobile 300 by the notification mode determination program. As shown in FIG. 7, the monitoring device 100 refers to the notification mode data and determines the notification mode according to the distance between the monitoring target and the vehicle 300. The notification mode data is stored in the storage unit 152. The notification mode A when the distance is 200 m to 500 m is, for example, a relatively quiet sound (or a low frequency vibration or a low sound), and the notification mode B when the distance is 100 m to 200 m is a louder sound (or a sound) than the general mode A. It is stipulated that the shorter the distance between the automobile 300 and the monitored object, the louder the sound (or the higher frequency vibration or the higher frequency sound), such as the vibration with a higher frequency or the sound with a higher frequency). ing.

The monitoring device 100 notifies the outside (driver 350) by the notification program in the notification mode determined by the notification mode determination program.

When the monitoring device 100 notifies the outside (driver 350) that the notification condition is satisfied, the monitoring program acquires position information from the photographing device 10 and tracks the position of the automobile 300. That is, when the monitoring device 100 notifies the outside that the notification condition is satisfied, the monitoring device 100 transmits a position information transmission request to the photographing device 10, and continuously receives the position information from the photographing device 10. It should be noted that the monitoring device 100 may not always carry out tracking after notification to the outside, but may carry out when a predetermined condition is satisfied. The predetermined condition is that, for example, the driver 350 inputs an instruction to start tracking to the monitoring device 100. When the monitoring device 100 receives the position information from the photographing device 10, the monitoring device 100 displays the position of the photographing device 10 (the position of the automobile 300) on the display screen 100a as shown in FIG. In the example of FIG. 8, it is shown on the display screen 100a that the image pickup device 10 has passed through the path R1 from the first position P1 and reached the current position P2. That is, when the notification condition is satisfied, the monitoring device 100 can track the position of the photographing device 10 (the position of the automobile 300), so that even if the automatic person 300 is stolen, it can be tracked.

Hereinafter, an operation example of the monitoring device 100 and the photographing device 10 will be described with reference to the flowchart of FIG. When the photographing device 10 acquires an image (step ST1) and transmits the image data (step ST2), the monitoring device 100 receives the image data (step ST3). The monitoring device 100 performs image recognition (step ST4), determines whether or not the notification condition is satisfied (step ST5), and if the notification condition is satisfied, determines the notification mode (step ST6) and notifies (step ST6). Step ST7). Subsequently, it is determined whether or not the predetermined condition (tracking condition) is satisfied (step ST8), and if the tracking condition is satisfied, the position information is requested from the photographing device 10 (step ST9), and the position information is transmitted from the photographing device 10. Is received (step ST10), and the photographing device 10 (vehicle 300) is tracked (step ST11).

In the above-described embodiment, the photographing device 10 acquires an image, and the monitoring device 100 receives the image data from the photographing device 10 and performs the processing after the image recognition. However, the notification to the driver 350 is described. Other processing may be performed by the photographing apparatus 10. In this case, the photographing device 10 is configured to transmit information indicating that the notification condition is satisfied to the monitoring device 100, and the monitoring device 100 notifies the outside, and the photographing device 10 and the monitoring device 100 are integrated into the monitoring device. Or, configure a monitoring system. Alternatively, the photographing apparatus 10 may be made to perform a part of the processing other than the notification to the driver 350. Also in this case, the photographing device 10 and the monitoring device 100 are integrated to form a monitoring device or a monitoring system. Further, the photographing device 10 is not limited to the drive recorder as long as it can photograph, and may be a mobile phone or a smartphone.

The monitoring device, monitoring system, monitoring method, and monitoring program of the present invention are not limited to the above embodiments, and various changes can be made without departing from the gist of the present invention.

10 Imaging device 100 Monitoring device 300 Automobile 352 Driver 360,362 Monitoring target

Claims (9)

A monitoring device that monitors the outside of a car
Image data acquisition means for acquiring image data of an object,
Among the objects included in the image data, an object recognition means for recognizing a monitoring target and
When the object recognition means recognizes the monitoring target, the notification condition determination means for determining whether or not the notification condition for notifying the outside is satisfied, and the notification condition determination means.
A notification means for notifying the outside of the condition of the notification condition by a predetermined notification mode,
A speed change recognizing means for recognizing a change in speed at which the monitored object approaches the automobile,
Have,
The notification condition is a monitoring device in which the speed has increased. The monitoring device according to claim 1, wherein the object recognition means refers to data generated by deep learning. It has a distance estimation means for estimating the distance between the monitored object and the automobile.
The notification condition is that the monitored object is located within a predetermined distance from the automobile.
The monitoring device according to claim 1 or 2. A notification mode determining means for determining a notification mode for notifying the outside based on the distance between the monitored object and the automobile.
The monitoring device according to claim 3. It has an approach determining means for determining whether or not the monitored object is approaching the automobile.
The notification condition is that the monitored object is close to the vehicle.
The monitoring device according to claim 1 or 2. The vehicle has a tracking means for acquiring position information indicating the position of the vehicle and tracking the vehicle when the notification means notifies the outside that the notification condition is satisfied by a predetermined notification mode.
The monitoring device according to claim 1 or 2. A monitoring system that monitors the outside of a car
An image data acquisition means for acquiring image data of an object photographed by the photographing means,
Among the objects included in the image data, an object recognition means for recognizing a monitoring target and
When the object recognition means recognizes the monitoring target, the notification condition determination means for determining whether or not the notification condition for notifying the outside is satisfied, and the notification condition determination means.
A notification means for notifying the outside of the condition of the notification condition by a predetermined notification mode,
A speed change recognizing means for recognizing a change in speed at which the monitored object approaches the automobile,
Have,
The notification condition is a monitoring system in which the speed has increased . A monitoring device that monitors the outside of the car
Image data acquisition step to acquire image data of an object,
Among the objects included in the image data, an object recognition step for recognizing a monitoring target and
When the monitoring target is recognized in the object recognition step, the notification condition determination step for determining whether or not the notification condition for notifying the outside is satisfied, and the notification condition determination step.
A notification step for notifying the outside of the condition of the notification condition by a predetermined notification mode,
A speed change recognition step for recognizing a change in speed at which the monitored object approaches the vehicle, and
Have,
The notification condition is a monitoring method in which the speed has increased. A control device that manages a monitoring device that monitors the outside of a car,
Image data acquisition means for acquiring image data of an object,
An object recognition means for recognizing a monitoring target among the objects included in the image data,
When the object recognition means recognizes the monitoring target, the notification condition determination means for determining whether or not the notification condition for notifying the outside is satisfied, and the notification condition determination means.
A notification means for notifying the outside of the condition of the notification condition by a predetermined notification mode,
A speed change recognizing means that recognizes a change in speed at which the monitored object approaches the automobile.
To function as
The notification condition is a monitoring program in which the speed has increased.

Images