JP7485449B2 - Monitoring device, monitoring method, and program - Google Patents

Description

The present invention relates to a monitoring device and a monitoring method for monitoring dangerous situations of people, such as falling down a step, and further to a program for implementing these.

If a person falls onto the tracks at a station platform, the train must be stopped immediately. For this reason, station staff monitor the situation by visual inspection or by checking security camera footage, but these measures alone are not sufficient. For this reason, monitoring systems that use image processing have been proposed in recent years (see, for example, Patent Documents 1 and 2).

Patent Document 1 discloses a system that uses a surveillance camera to monitor people on the platform and trains entering the platform. The system disclosed in Patent Document 1 determines, from images from the surveillance camera, whether a person is present in a set area on the platform, and further whether the person has come into contact with a train.

If the system disclosed in Patent Document 1 determines that a person is present in the set area, it outputs an alarm from a speaker on the platform to alert the person. If the system disclosed in Patent Document 1 determines that a person has come into contact with a train, it outputs a train stop signal to the train.

Furthermore, Patent Document 2 discloses a monitoring system that uses a stereo camera to detect people falling from the tracks onto the platform. In the system disclosed in Patent Document 2, multiple stereo cameras are prepared and placed at intervals along the edge of the platform. These multiple stereo cameras capture the entire edge of the platform thoroughly.

The system disclosed in Patent Document 2 detects a person standing on the edge of the platform when it captures an image of the person with one of the stereo cameras, and then uses the two images obtained from the stereo cameras to calculate the distance of the detected person from the camera, which is used as distance information. The system disclosed in Patent Document 1 then uses the images and distance information to track the person standing on the edge of the platform, and issues an alarm if the person gets too close to the edge of the platform, and sends a stop signal to the train if it detects a fall.

JP 2001-341642 A JP 2004-058737 A

However, the system disclosed in Patent Document 1 only determines whether a person is present in a set area on the platform. For this reason, the system disclosed in Patent Document 1 may not be able to detect a person who is about to fall off the platform, that is, if most of the person's body is outside the set area. In addition, the system disclosed in Patent Document 1 is only intended for stations where there are no station staff on the platform, and will erroneously detect station staff if they are monitoring the edge of the platform.

On the other hand, the system disclosed in Patent Document 2 can use images and distance information to track people at the edge of the platform, which is believed to solve the problems with the system disclosed in Patent Document 1.

However, because the viewing angle of a stereo camera is narrow, a large number of stereo cameras are required to realize the system disclosed in Patent Document 2. In addition, the cost of the stereo camera itself is higher than the cost of a monocular camera. For this reason, the system disclosed in Patent Document 2 has the problem of a significant increase in cost.

An example of an object of the present invention is to provide a monitoring device, a monitoring method, and a program that can solve the above problems and accurately detect a dangerous state while suppressing an increase in costs associated with the system.

In order to achieve the above object, a monitoring device according to one aspect of the present invention comprises:
an object detection unit that detects an object present in a monitoring area from an image of the monitoring area and sets a frame surrounding the detected object on the image;
a partial safety area extraction unit that compares a boundary line that separates a safety area and a non-safety area on the image with the frame, and when the boundary line crosses the frame, extracts an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination processing unit that extracts a portion corresponding to the partial safety area from a reference image in the monitoring area, calculates a difference between the extracted portion and the partial safety area, and determines whether the target is safe based on the calculated difference value;
The present invention is characterized in that it is equipped with:

In order to achieve the above object, a monitoring method according to one aspect of the present invention includes:
a setting step of detecting an object present in a monitoring area from an image of the monitoring area and setting a frame surrounding the detected object on the image;
an extraction step of comparing a boundary line separating a safety area and a non-safety area on the image with the frame, and extracting, when the boundary line crosses the frame, an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination step of extracting a portion corresponding to the partial safety area from a reference image in the monitoring area, calculating a difference between the extracted portion and the partial safety area, and determining whether or not the target is safe based on the calculated difference value;
The present invention is characterized in that it has the following features:

Furthermore, in order to achieve the above object, a program according to one aspect of the present invention comprises:
On the computer,
a setting step of detecting an object present in a monitoring area from an image of the monitoring area and setting a frame surrounding the detected object on the image;
an extraction step of comparing a boundary line separating a safety area and a non-safety area on the image with the frame, and extracting, when the boundary line crosses the frame, an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination step of extracting a portion corresponding to the partial safety area from a reference image in the monitoring area, calculating a difference between the extracted portion and the partial safety area, and determining whether or not the target is safe based on the calculated difference value;
A program to execute .

As described above, the present invention makes it possible to accurately detect dangerous conditions while minimizing increases in system costs.

FIG. 1 is a block diagram showing a schematic configuration of a monitoring device according to an embodiment of the present invention. FIG. 2 is a block diagram showing in more detail the configuration of the monitoring device according to the embodiment of the present invention. 3A and 3B are diagrams showing an example of a target detection process according to an embodiment of the present invention. 4A and 4B are diagrams showing an example of a difference calculation process according to an embodiment of the present invention. FIG. 5 is a flow chart showing the operation of the monitoring device according to the embodiment of the present invention. FIG. 6 is a block diagram showing an example of a computer that realizes a monitoring device according to an embodiment of the present invention.

(Embodiment)
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A monitoring device, a monitoring method, and a program according to an embodiment of the present invention will be described below with reference to FIGS.

[Device configuration]
First, the schematic configuration of a monitoring device according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the schematic configuration of a monitoring device according to an embodiment of the present invention.

The monitoring device 10 in this embodiment shown in Fig. 1 is a device for monitoring dangerous situations of people, such as a person falling down a step. As shown in Fig. 1, the monitoring device 10 includes an object detection unit 11, a partial safety area extraction unit 12, and a judgment processing unit 13.

The object detection unit 11 detects objects present in the monitoring area from the image of the monitoring area and sets a frame around the detected object on the image. An example of the object is a person. An example of the monitoring area is a train station platform.

The partial safety area extraction unit 12 compares the boundary line that separates the safety area from the non-safety area on the image with the frame. Furthermore, when the boundary line crosses the frame, the partial safety area extraction unit 12 extracts the area that includes the safety area and is surrounded by the outer edge of the frame and the boundary line as the partial safety area.

The judgment processing unit 13 extracts a portion corresponding to the partial safety area from the reference image in the monitoring area, calculates the difference between the extracted portion and the partial safety area, and judges whether the target is safe based on the calculated difference value.

In this way, in this embodiment, a judgment is made based on the difference between the partial safety area and the corresponding area of the reference image, so a dangerous state can be reliably detected even if most of the target person's body is outside the safety area. In addition, because distance information is not required for the judgment, an increase in the cost of the system is suppressed. In other words, according to this embodiment, a dangerous state can be accurately detected while suppressing an increase in the cost of the system.

Next, the configuration and functions of the monitoring device 10 in this embodiment will be described in more detail with reference to Figures 2 to 4. Figure 2 is a block diagram showing in more detail the configuration of the monitoring device in this embodiment of the present invention.

As shown in Figure 2, in this embodiment, the monitoring device 10 is connected to an imaging device 20 and a display device 30. The imaging device 20 is a surveillance camera, and outputs image data of a captured image of a surveillance area at a set frame rate. The display device 30 is a liquid crystal display device or the like, and displays the surveillance image, the judgment results by the monitoring device 10, etc. on the screen.

2, the monitoring device 10 includes an image data acquisition unit 14 and a display unit 15 in addition to the above-mentioned object detection unit 11, partial safety area extraction unit 12, and judgment processing unit 13. The image data acquisition unit 14 acquires image data output from the imaging device 20. The image data acquisition unit 14 also outputs the acquired image data to the object detection unit 11 and the display unit 15. When the image data is output, the display unit 15 sends the image data to the display device 30, which displays the captured image.

In this embodiment, when a person to be detected is present in the image data, the object detection unit 11 detects the person and sets a rectangular frame surrounding the person. Specifically, the object detection unit 11 identifies an area having features indicative of a person while scanning the image data. Since this identified area corresponds to a person, the object detection unit 11 sets a rectangular frame surrounding the identified area. In addition, the features representing a person are obtained in advance, for example, by performing machine learning using images containing people as training data.

Figures 3(a) and 3(b) are diagrams showing an example of an object detection process in an embodiment of the present invention. As shown in Figures 3(a) and (b), a boundary line 41 is set in an image 40, separating a safe area A from a non-safe area B. In the example of Figures 3(a) and (b), the monitored area is a station platform. The boundary line 41 is set along the edge of the platform, with the platform area being the safe area A and the area where the tracks are present being the non-safe area B.

In the example of Fig. 3(a), the target detection unit 11 detects people 51 and 52 as targets, and sets frames 53 and 54 for them, respectively. In the example of Fig. 3(b), the target detection unit 11 detects people 55 and 56 as targets, and sets frames 57 and 58 for them, respectively.

In this embodiment, the partial safety area extraction unit 12 compares the boundary line 41 with each frame set for each person and determines whether or not a frame exists that is crossed by the boundary line 41. If a frame exists that is crossed by the boundary line 41, the partial safety area extraction unit 12 extracts an area that includes a safety area and is surrounded by the outer edge of this frame and the boundary line as a partial safety area. The partial safety area extraction unit 12 also identifies the coordinates of the extracted partial safety area and sends the identified coordinates to the judgment processing unit 13.

Specifically, in the example of Fig. 3(a) described above, the partial safety area extraction unit 12 determines that the boundary line 41 crosses the frame 54 set for the person 52 standing at the edge of the platform, and extracts the hatched area a1 as the partial safety area. Also, in the example of Fig. 3(b), the partial safety area extraction unit 12 determines that the boundary line 41 crosses the frame 58 set for the person 56 standing at the edge of the platform, and extracts the hatched area a2 as the partial safety area.

In this embodiment, the judgment processing unit 13 first acquires a reference image and uses the coordinates of the partial safety area to extract from the reference image a portion that corresponds to the partial safety area (hereinafter referred to as the "portion corresponding to the partial safety area"). The reference image is created, for example, by calculating the average value of the pixel value for each pixel from multiple images obtained by photographing the monitored area over a certain period of time, and setting the calculated average value as the pixel value of each pixel. In addition, an image of the monitored area in which no people are present may be used as the reference image.

Furthermore, the judgment processing unit 13 calculates the difference between the partial safety area and the extracted portion of the partial safety area, as shown in Figures 4(a) and 4(b), respectively. Figures 4(a) and 4(b) are diagrams showing an example of a difference calculation process in an embodiment of the present invention. Figure 4(a) shows a difference calculation process for the partial safety area shown in Figure 3(a), and Figure 4(b) shows a difference calculation process for the partial safety area shown in Figure 3(b).

In this embodiment, the judgment processing unit 13 then judges whether the difference is greater than or equal to a threshold value. If the result of the judgment is that the difference is greater than or equal to the threshold value, most of the person is in the safe area, and so the judgment processing unit 13 judges that it is safe. On the other hand, if the result of the judgment is that the difference is less than the threshold value, most of the person is not in the safe area, and so the judgment processing unit 13 judges that it is not safe. Furthermore, the judgment processing unit 13 sends the result of the judgment to the display unit 15.

When the display unit 15 receives the judgment result, it creates an image showing the judgment result and sends the image data of the created image to the display device 30. As a result, the judgment result is displayed on the display screen of the display device 30. For example, if the display device 30 is installed in a station control room or the like, station staff can immediately know of unsafe situations. As a result, safety is ensured even more reliably during railway operations.

[Device Operation]
Next, the operation of the monitoring device 10 in the embodiment of the present invention will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing the operation of the monitoring device in the embodiment of the present invention. In the following description, Figs. 1 to 4 will be referred to as appropriate. Also, in this embodiment, the monitoring method is implemented by operating the monitoring device. Therefore, the description of the monitoring method in this embodiment will be replaced by the following description of the operation of the monitoring device 10.

5, first, the image data acquisition unit 14 acquires image data output from the imaging device 20 (step A1). The image data acquisition unit 14 then outputs the acquired image data to the object detection unit 11 and the display unit 15.

Next, the object detection unit 11 determines whether or not a person, which is the detection target, is present in the image data acquired in step A1 (step A2). Specifically, in step A2, the object detection unit 11 scans the image of the image data and determines whether or not an area having a feature amount indicating a person is present.

If the result of the determination in step A2 is that no person is present, step A1 is executed again. On the other hand, if the result of the determination in step A2 is that a person is present, the object detection unit 11 sets a rectangular frame surrounding the present person (step A3).

Next, the partial safety area extraction unit 12 compares the boundary line 41 with each frame set in step A3, and determines whether or not a frame exists that is crossed by the boundary line 41 (step A4). If the result of the determination in step A4 is that no frame exists that is crossed by the boundary line 41, step A1 is executed again. On the other hand, if the result of the determination in step A4 is that a frame exists that is crossed by the boundary line 41, the partial safety area extraction unit 12 extracts a partial safety area from this frame (step A5).

Next, the determination processing unit 13 extracts the portion of the partial safety area from the reference image using the coordinates of the partial safety area extracted in step A5 (step A6). Furthermore, the determination processing unit 13 calculates the difference between the partial safety area extracted in step A5 and the portion of the partial safety area extracted in step A6 (step A7).

Next, the judgment processing unit 13 judges whether the difference calculated in step A7 is equal to or greater than a threshold value (step A8). If the result of the judgment in step A8 is that the difference is equal to or greater than the threshold value, the judgment processing unit 13 judges that it is safe (step A9). On the other hand, if the result of the judgment is that the difference is less than the threshold value, the judgment processing unit 13 judges that it is not safe (step A10).

When step A9 or A10 is executed, the judgment processing unit 13 sends the judgment result to the display unit 15. As a result, the display unit 15 creates an image showing the judgment result and outputs image data of the created image to the display device 30 (step A11). As a result, the judgment result is displayed on the display screen of the display device 30.

Once step A11 is performed, step A1 is performed again. Steps A1 to A11 are repeatedly performed as long as the monitoring device 10 is operating. Furthermore, steps A1 to A11 may be performed for each single frame, or for each two or more frames.

[Effects of the embodiment]
As described above, according to this embodiment, when the monitoring area is, for example, a station platform, it is possible to accurately detect a person who is about to fall off the tracks. Since distance information is not required to detect a fall, an increase in the cost of the system is suppressed. Furthermore, according to this embodiment, station staff can immediately know from the screen of the display device 30 that a person is about to fall off the platform, so safety is more reliably ensured during railway operations.

[Modification]
In the above example, the case where the monitored area is a station platform has been described, but in this embodiment, the monitored area may be something other than a station platform. For example, the monitored area may be a production line in a factory. In this case, the monitoring device 10 can also determine whether or not a person has entered a no-entry area (around a machine, on a conveyor, etc.) set in the factory.

In addition, in this embodiment, in addition to calculating the difference between the portion of the reference image corresponding to the partial safety area and the partial safety area, or instead of calculating the difference, the judgment processing unit 13 can also calculate the similarity between the portion of the reference image corresponding to the partial safety area and the partial safety area. In this case, the judgment processing unit 13 judges whether the target person is safe based on the calculated similarity value. Specifically, if the similarity is low, the judgment processing unit 13 judges that the target person is safe, and if the similarity is high, the judgment processing unit 13 judges that the target person is unsafe.

[program]
The program in this embodiment may be any program that causes a computer to execute steps A1 to A11 shown in Fig. 5. By installing and executing this program in a computer, the monitoring device 10 and monitoring method in this embodiment can be realized. In this case, the processor of the computer functions as the object detection unit 11, the partial safe area extraction unit 12, the judgment processing unit 13, the image data acquisition unit 14, and the display unit 15, and performs processing.

The program in this embodiment may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as any one of the target detection unit 11, partial safety area extraction unit 12, judgment processing unit 13, image data acquisition unit 14, and display unit 15.

Here, a computer that realizes the monitoring device 10 by executing the program in this embodiment will be described with reference to Figure 6. Figure 6 is a block diagram showing an example of a computer that realizes the monitoring device in this embodiment of the present invention.

6, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader/writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. The computer 110 may also include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111.

The CPU 111 loads the program (group of codes) in this embodiment stored in the storage device 113 into the main memory 112 and executes the codes constituting the program in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). The program in this embodiment is provided in a state stored in a computer-readable recording medium 120. The program in this embodiment may be distributed over the Internet connected via the communication interface 117.

Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader/writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes the results of processing in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

Specific examples of recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or optical recording media such as a CD-ROM (Compact Disk Read Only Memory).

In addition, the monitoring device 10 in this embodiment can be realized by using hardware corresponding to each part, rather than a computer on which a program is installed. Furthermore, the monitoring device 10 may be realized in part by a program and the remaining part by hardware.

Some or all of the above-described embodiments can be expressed by (Appendix 1) to (Appendix 6) described below, but are not limited to the following descriptions.

(Appendix 1)
an object detection unit that detects an object present in a monitoring area from an image of the monitoring area and sets a frame surrounding the detected object on the image;
a partial safety area extraction unit that compares a boundary line that separates a safety area and a non-safety area on the image with the frame, and when the boundary line crosses the frame, extracts an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination processing unit that extracts a portion corresponding to the partial safety area from a reference image in the monitoring area, calculates a difference between the extracted portion and the partial safety area, and determines whether the target is safe based on the calculated difference value;
A monitoring device comprising:

(Appendix 2)
2. The monitoring device of claim 1,
The determination processing unit calculates a similarity between the extracted portion and the partial safety region in addition to or instead of calculating the difference, and determines whether the target is safe based on the calculated similarity value.
A monitoring device comprising:

(Appendix 3)
a setting step of detecting an object present in a monitoring area from an image of the monitoring area and setting a frame surrounding the detected object on the image;
an extraction step of comparing a boundary line separating a safety area and a non-safety area on the image with the frame, and extracting, when the boundary line crosses the frame, an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination step of extracting a portion corresponding to the partial safety area from a reference image in the monitoring area, calculating a difference between the extracted portion and the partial safety area, and determining whether or not the target is safe based on the calculated difference value;
A monitoring method comprising:

(Appendix 4)
4. The monitoring method according to claim 3, further comprising:
In the determination step, in addition to or instead of calculating the difference, a similarity between the extracted portion and the partial safety region is calculated, and whether or not the object is safe is determined based on the calculated similarity value.
A monitoring method comprising:

(Appendix 5)
On the computer,
a setting step of detecting an object present in a monitoring area from an image of the monitoring area and setting a frame surrounding the detected object on the image;
an extraction step of comparing a boundary line separating a safety area and a non-safety area on the image with the frame, and extracting, when the boundary line crosses the frame, an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination step of extracting a portion corresponding to the partial safety area from a reference image in the monitoring area, calculating a difference between the extracted portion and the partial safety area, and determining whether or not the target is safe based on the calculated difference value;
A program that executes .

(Appendix 6)
6. The program according to claim 5,
In the determination step, in addition to or instead of calculating the difference, a similarity between the extracted portion and the partial safety region is calculated, and whether or not the object is safe is determined based on the calculated similarity value.
A program characterized by:

Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2019-136637, filed on July 25, 2019, the disclosure of which is incorporated herein in its entirety.

As described above, the present invention makes it possible to accurately detect dangerous conditions while suppressing increases in system costs. The present invention is useful for ensuring safety, for example, on train platforms, in factories, etc.

REFERENCE SIGNS LIST 10 Monitoring device 11 Object detection unit 12 Partial safe area extraction unit 13 Judgment processing unit 14 Image data acquisition unit 15 Display unit 20 Imaging device 30 Display device 110 Computer 111 CPU
112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader/writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims (6)

an object detection unit that detects an object present in a monitoring area from an image of the monitoring area and sets a frame surrounding the detected object on the image;
a partial safety area extraction unit that compares a boundary line that separates a safety area and a non-safety area on the image with the frame, and when the boundary line crosses the frame, extracts an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination processing unit that extracts a portion corresponding to the partial safety area from a reference image in the monitoring area, calculates a difference between the extracted portion and the partial safety area, and determines whether the target is safe based on the calculated difference value;
Equipped with
The reference image is an image created by calculating an average value of pixel values for each pixel from a plurality of images obtained by photographing the monitoring area for a certain period of time, and setting the calculated average value as the pixel value of each pixel.
A monitoring device comprising: The monitoring device according to claim 1,
The determination processing unit calculates a similarity between the extracted portion and the partial safety region in addition to or instead of calculating the difference, and determines whether the target is safe based on the calculated similarity value.
A monitoring device comprising: a setting step of detecting an object present in a monitoring area from an image of the monitoring area and setting a frame surrounding the detected object on the image;
an extraction step of comparing a boundary line separating a safety area and a non-safety area on the image with the frame, and extracting, when the boundary line crosses the frame, an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination step of extracting a portion corresponding to the partial safety area from a reference image in the monitoring area, calculating a difference between the extracted portion and the partial safety area, and determining whether or not the target is safe based on the calculated difference value;
having
The reference image is an image created by calculating an average value of pixel values for each pixel from a plurality of images obtained by photographing the monitoring area for a certain period of time, and setting the calculated average value as the pixel value of each pixel.
A monitoring method comprising: 4. The monitoring method according to claim 3, further comprising:
In the determination step, in addition to or instead of calculating the difference, a similarity between the extracted portion and the partial safety region is calculated, and whether or not the object is safe is determined based on the calculated similarity value.
A monitoring method comprising: On the computer,
a setting step of detecting an object present in a monitoring area from an image of the monitoring area and setting a frame surrounding the detected object on the image;
an extraction step of comparing a boundary line separating a safety area and a non-safety area on the image with the frame, and extracting, when the boundary line crosses the frame, an area that includes the safety area and is surrounded by an outer edge of the frame and the boundary line as a partial safety area;
a determination step of extracting a portion corresponding to the partial safety area from a reference image in the monitoring area, calculating a difference between the extracted portion and the partial safety area, and determining whether or not the target is safe based on the calculated difference value;
Run the command ,
The reference image is an image created by calculating an average value of pixel values for each pixel from a plurality of images obtained by photographing the monitoring area for a certain period of time, and setting the calculated average value as the pixel value of each pixel.
program. The program according to claim 5,
In the determination step, in addition to or instead of calculating the difference, a similarity between the extracted portion and the partial safety region is calculated, and whether or not the target is safe is determined based on the calculated similarity value.
A program characterized by:

Images