JP7550640B2 - Railway line monitoring device and railway line monitoring system - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applied. Published on the website on August 5, 2020 as a proceedings collection for the 2020 National Conference of the Japan Society of Civil Engineers.

The present invention relates to a wayside monitoring device and a wayside monitoring system that use a processor to execute processing to detect abnormalities around the track based on camera images output from cameras installed along railway tracks.

In railways, abnormalities around the tracks, such as falling rocks, landslides, avalanches, and other obstacles on the tracks, can sometimes hinder the safe passage of trains. For this reason, in order to quickly respond to abnormalities around the tracks, wayside monitoring systems have become widespread, which allow remote monitoring of abnormalities around the tracks by taking pictures of the area with cameras installed along the railway line.

In such a wayside monitoring system, a monitor can visually check for abnormalities around the track by viewing camera images, but the burden on the monitor can be reduced by providing a monitoring device that detects abnormalities around the track and alerting the monitor when an abnormality around the track is detected.

A conventional monitoring device for detecting such abnormalities around the track is known to use technology that compares real-time camera images of the monitored area with camera images of normal conditions and detects abnormalities based on significant changes that appear in the real-time camera images (see Patent Document 1).

JP 2004-291935 A

Now, even when there are no abnormalities around the track, i.e. obstacles on the track such as falling rocks, mudslides, avalanches, etc. that impede the safe passage of trains, there are many phenomena that cause significant changes in the camera image. For example, significant changes appear in the camera image when the camera rotates or performs PTZ movements (pan, tilt, zoom), when a train passes over the track, when there is heavy rain or snowfall, when the light from a train or automobile is shining on the camera, or when an object such as an insect is floating near the camera lens.

However, with conventional anomaly detection methods, when an event like the one described above occurs and a major change appears in the camera image, a false positive occurs, and a false alarm is sent to the observer. If these false alarms occur frequently, the task of checking the alarm becomes a heavy burden for the observer.

The present invention has as its main objective the provision of a wayside monitoring device and wayside monitoring system that can suppress false positives in anomaly detection processing by excluding camera images that are highly likely to cause false positives in the anomaly detection processing from the targets of the anomaly detection processing in advance.

The wayside monitoring device of the present invention is a wayside monitoring device that uses a processor to execute an abnormality detection process that detects abnormalities around a track based on camera images taken by cameras installed along a railway line, wherein the processor sets a target area for the abnormality detection process on the camera image based on a user's operational input, performs the abnormality detection process to detect abnormalities in the target area based on the real-time camera image output from the camera, and further performs an exclusion determination process prior to the abnormality detection process to determine whether or not the real-time camera image should be excluded from the target of the abnormality detection process, and when an abnormality in the target area is detected, outputs notification information to notify a monitor of the occurrence of the abnormality, while generating and outputting a surveillance camera image in which a status image representing one of a normal status, an abnormal status, or an excluded status as the status of the target area is superimposed on the camera image depending on the detection result of the abnormality detection process and the determination result of the exclusion determination process .

Furthermore, the wayside monitoring device of the present invention is a wayside monitoring device that uses a processor to execute an abnormality detection process to detect abnormalities around the track based on camera images taken by cameras installed along the railway line, and the processor is configured to set a target area for the abnormality detection process on the camera image based on a user's operational input, perform the abnormality detection process to detect abnormalities in the target area based on the real-time camera image output from the camera, and further perform an exclusion determination process prior to the abnormality detection process to determine whether or not the real-time camera image should be excluded from the target of the abnormality detection process, and generate and output a surveillance camera image in which a status image representing one of a normal status, an abnormal status, or an excluded status as the status of the target area is superimposed on the camera image depending on the detection result of the abnormality detection process and the determination result of the exclusion determination process .

Furthermore, the trackside monitoring system of the present invention is a trackside monitoring system comprising cameras installed along a railway line and an information processing device which executes an abnormality detection process to detect abnormalities around the track based on camera images captured by the cameras, wherein the information processing device sets a target area for the abnormality detection process on the camera images based on operational input from a user, and performs the abnormality detection process to detect abnormalities in the target area based on the real-time camera images output from the camera, and further performs an exclusion determination process prior to the abnormality detection process to determine whether or not the real-time camera images should be excluded from the target of the abnormality detection process , and when an abnormality in the target area is detected, outputs notification information to notify a monitor of the occurrence of the abnormality, while generating and outputting a surveillance camera image in which a status image representing one of a normal status, an abnormal status, or an excluded status as the status of the target area is superimposed on the camera image depending on the detection result of the abnormality detection process and the determination result of the exclusion determination process.

Furthermore, the trackside monitoring system of the present invention is a trackside monitoring system comprising cameras installed along a railway line and an information processing device which executes an abnormality detection process to detect abnormalities around the track based on camera images captured by the cameras, wherein the information processing device sets a target area for the abnormality detection process on the camera images based on user operation input, performs the abnormality detection process to detect abnormalities in the target area based on the real-time camera images output from the camera, and further performs an exclusion determination process prior to the abnormality detection process to determine whether or not the real-time camera images should be excluded from the target of the abnormality detection process, and generates and outputs a surveillance camera image in which a status image representing one of a normal status, an abnormal status, or an excluded status as the status of the target area is superimposed on the camera image depending on the detection result of the abnormality detection process and the determination result of the exclusion determination process .

According to the present invention, a surveillance camera image is output on which a status image representing either a normal state, an abnormal state, or an exclusion state as the state of the target area is superimposed depending on the detection result of the anomaly detection process and the judgment result of the exclusion judgment process, thereby allowing a monitor to instantly grasp the detection result of the anomaly detection and the judgment result of the exclusion judgment .

Overall configuration of a railway line monitoring system according to the present embodiment. FIG. 13 is an explanatory diagram showing a monitoring screen displayed on a monitoring terminal 4. A block diagram showing a schematic configuration of a monitoring server 2. FIG. 1 is an explanatory diagram showing an overview of a process performed by a monitoring server 2. FIG. 2 is an explanatory diagram showing the timing of display processing and notification processing according to the detection result of the abnormality detection processing and the judgment result of the exclusion judgment processing performed by the monitoring server 2. FIG. 13 is an explanatory diagram showing an area setting screen displayed on the display unit of the monitoring server 2. FIG. 1 is an explanatory diagram showing an outline of the exclusion determination process performed by the monitoring server 2. FIG. 2 is an explanatory diagram showing an overview of an abnormality detection process performed by the monitoring server 2. Flow diagram showing the operation procedure of the monitoring server 2.

A first invention made to solve the above-mentioned problems is a wayside monitoring device in which a processor executes an abnormality detection process to detect abnormalities around a track based on camera images taken by a camera installed along a railway line, wherein the processor sets a target area for the abnormality detection process on the camera image based on a user's operational input, and performs the abnormality detection process to detect abnormalities in the target area based on the real-time camera image output from the camera, and further performs an exclusion determination process prior to the abnormality detection process to determine whether or not the real-time camera image should be excluded from the target of the abnormality detection process, and when an abnormality in the target area is detected, outputting notification information to notify a monitor of the occurrence of the abnormality, while generating and outputting a surveillance camera image in which a status image representing one of a normal status, an abnormal status, or an excluded status as the status of the target area is superimposed on the camera image depending on the detection result of the abnormality detection process and the determination result of the exclusion determination process .

According to this , a surveillance camera image is output on which a status image representing either a normal state, an abnormal state, or an exclusion state as the state of the target area is superimposed depending on the detection results of the anomaly detection process and the judgment results of the exclusion judgment process, thereby enabling the monitor to instantly grasp the detection results of the anomaly detection and the judgment results of the exclusion judgment .

In addition, in the second invention, the processor is configured to detect an abnormality in the target area using a classifier that has been trained in advance using the camera images in an abnormal state and the camera images in a normal state as the abnormality detection process.

This allows for accurate anomaly detection processing.

In addition, the third invention is configured such that, as the exclusion determination process, the processor acquires subject motion information based on the camera image, and determines, based on the motion information, whether or not to exclude the camera image from the anomaly detection process.

This makes it possible to exclude camera images in a state where there is a high possibility of false positives occurring in the anomaly detection process from the target of the anomaly detection process. For example, false positives occur when the camera is rotating or performing a PTZ operation, when a train is passing on the track, when poor images are occurring due to heavy rainfall or snowfall, when poor images are occurring due to illumination by train or automobile headlights, or when poor images are occurring due to an object such as an insect floating near the camera, and camera images capturing such events can be excluded from the target of the anomaly detection process.

In addition, the fourth invention is configured such that, when an exclusion state in which a determination result is obtained that excludes the camera image from the target of the anomaly detection process continues for a predetermined time or more, the processor sets the camera image to a continuous exclusion state and outputs the notification information indicating that.

This allows the monitor to know if an exclusion state, i.e., a state in which anomaly detection processing is not being performed, has continued for a long period of time.

In addition, in the fifth invention, when a determination result is obtained that the camera image should not be excluded from the anomaly detection processing and the continuous exclusion state is released, the processor outputs the notification information indicating that fact.

This allows the monitor to know that the exclusion state has been lifted and the system has returned to a state in which anomaly detection processing can be performed.

Further, a sixth invention is a wayside monitoring device in which a processor executes an abnormality detection process to detect abnormalities around a track based on camera images taken by a camera installed along a railway line, wherein the processor sets a target area of the abnormality detection process on the camera image based on a user's operational input, performs the abnormality detection process to detect abnormalities in the target area based on the real-time camera image output from the camera, and further performs an exclusion determination process prior to the abnormality detection process to determine whether or not the real-time camera image should be excluded from the target of the abnormality detection process, and generates and outputs a surveillance camera image in which a status image representing one of a normal status, an abnormal status, or an excluded status as the status of the target area is superimposed on the camera image depending on the detection result of the abnormality detection process and the determination result of the exclusion determination process .

With this, a surveillance camera image is output on which a status image representing either a normal state, an abnormal state, or an exclusion state as the state of the target area is superimposed depending on the detection results of the anomaly detection process and the judgment results of the exclusion judgment process, allowing the monitor to instantly grasp the detection results of the anomaly detection and the judgment results of the exclusion judgment.

In a seventh aspect of the present invention, the processor is configured to change a display form of a frame image indicating the target area as the status image .

This allows the observer to immediately grasp the detection results of the anomaly detection process and the judgment results of the exclusion judgment process based on the display format of the frame image superimposed on the camera image.

An eighth invention is a wayside monitoring system comprising cameras installed along a railway line and an information processing device which executes an abnormality detection process for detecting abnormalities around the track based on camera images captured by the cameras, wherein the information processing device sets a target area for the abnormality detection process on the camera image based on a user's operational input, performs the abnormality detection process for detecting abnormalities in the target area based on the real-time camera image output from the camera, and further performs an exclusion determination process prior to the abnormality detection process for determining whether or not the real-time camera image should be excluded from the target of the abnormality detection process, and when an abnormality in the target area is detected, outputs notification information for notifying a monitor of the occurrence of the abnormality, while generating and outputting a surveillance camera image in which a status image representing one of a normal status, an abnormal status, or an excluded status as the status of the target area is superimposed on the camera image depending on the detection result of the abnormality detection process and the determination result of the exclusion determination process .

According to this, similarly to the first aspect of the invention, the supervisor can immediately grasp the detection result of the abnormality detection and the determination result of the exclusion determination .

A ninth invention is a wayside monitoring system comprising cameras installed along a railway line and an information processing device which executes an abnormality detection process to detect abnormalities around the track based on camera images captured by the cameras, wherein the information processing device sets a target area for the abnormality detection process on the camera image based on a user's operational input, performs the abnormality detection process to detect abnormalities in the target area based on the real-time camera image output from the camera, and further performs an exclusion determination process prior to the abnormality detection process to determine whether or not the real-time camera image should be excluded from the target of the abnormality detection process, and generates and outputs a surveillance camera image in which a status image representing one of a normal status, an abnormal status, or an excluded status as the status of the target area is superimposed on the camera image depending on the detection result of the abnormality detection process and the determination result of the exclusion determination process .

According to this, similarly to the first aspect of the invention, the supervisor can immediately grasp the detection result of the abnormality detection and the determination result of the exclusion determination.

The following describes an embodiment of the present invention with reference to the drawings.

First Embodiment
FIG. 1 is a diagram showing the overall configuration of a railway line monitoring system according to this embodiment.

This wayside monitoring system includes a camera 1, a monitoring server 2 (wayside monitoring device, information processing device), a recorder 3 (camera image storage device), a monitoring terminal 4 (camera image viewing device), a warning light 5 (alarm device), and a user terminal 6. The camera 1, the monitoring server 2, the recorder 3, the monitoring terminal 4, the warning light 5, and the user terminal 6 are connected via a network.

Camera 1 is installed along the railway line and captures images of the monitoring area around the track. Camera 1 also transmits the camera images to monitoring server 2.

The monitoring server 2 detects abnormalities around the track, i.e., conditions that impede the safe passage of trains due to obstacles on the track, such as falling rocks, mudslides, avalanches, etc., by image analysis of camera images taken of the monitoring area (anomaly detection process). In addition, prior to the anomaly detection process, the monitoring server 2 determines whether or not to exclude the camera image from the anomaly detection process by image analysis of the camera image (exclusion determination process).

When the monitoring server 2 detects an abnormality around the track, it generates and outputs notification information to notify the monitor of the abnormality. In this embodiment, the notification information is sent to the user terminal 6 as an email containing a message about the abnormality. Here, the notification information is not limited to email, and may be sent to a pre-specified user terminal 6 using a push function. Also, as the notification information, control information is sent to the warning light 5 to instruct the warning light 5 to perform a notification operation (lighting or blinking).

The monitoring server 2 also sets a target area on the camera image capturing the monitored area in response to a specified operation by the administrator (user). The monitoring server 2 then generates a monitoring camera image in which the frame line of the target area is combined with the camera image, and transmits the generated image to the recorder 3. At this time, the frame line of the target area is drawn in a display form according to the detection results of the anomaly detection process and the judgment results of the exclusion judgment process. The monitoring server 2 also transmits the time of the alert and the identification information of the camera 1 as alert information to the recorder 3.

The recorder 3 stores the surveillance camera images received from the surveillance server 2. The recorder 3 also stores the alert information (alert time, camera 1 identification information, etc.) received from the surveillance server 2 in association with the surveillance camera images. Note that in the above example, the camera images are acquired from the surveillance server 2 by the recorder 3, but the camera images may also be stored in the recorder 3 directly from the camera 1.

The monitoring terminal 4 is installed in the monitoring room. In this monitoring terminal 4, the monitoring camera images acquired from the recorder 3 are displayed on the monitoring screen. This allows the monitor to specifically check the state of the monitored area. At this time, the monitoring terminal 4 acquires the real-time monitoring camera images generated by the monitoring server 2 via the recorder 3, allowing the monitor to view the real-time monitoring camera images. In addition, the monitoring terminal 4 acquires from the recorder 3 the past monitoring camera images stored in the recorder 3, allowing the monitor to view the past monitoring camera images.

The warning light 5 is installed in the monitoring room. This warning light 5 performs a predetermined notification operation (lighting or blinking) based on control information transmitted from the monitoring server 2. This allows a monitor in the monitoring room to recognize that an abnormality has been detected around the track in the monitoring area. Note that an alarm device that outputs a sound notifying of an abnormality around the track may be used together with or instead of the warning light 5.

The user terminal 6 is composed of a tablet, smartphone, or the like carried by the monitor. This user terminal 6 receives emails sent from the monitoring server 2. As a result, even if the monitor is not in the monitoring room, he or she can recognize that an abnormality has been detected around the track by viewing the anomaly detection message contained in the email. After viewing the anomaly detection message, the monitor can also confirm the specific circumstances of the abnormality by viewing the monitoring camera images on the monitoring terminal 4.

The surveillance camera image at the time of the alarm may be attached to the e-mail. Also, link information for obtaining the surveillance camera image from the recorder 3 may be attached to the e-mail. This allows the monitor to view the surveillance camera image on the user terminal 6.

Next, the monitoring screen displayed on the monitoring terminal 4 will be described. FIG. 2 is an explanatory diagram showing the monitoring screen. When the monitor specifies multiple cameras 1, the camera images corresponding to the specified multiple cameras 1 are displayed side by side on the monitoring screen.

Figure 2 (A) is a monitoring screen in a state where no abnormality is detected in the target area in the camera image (normal state). Figure 2 (B) is a monitoring screen in a state where an abnormality is detected in the target area in the camera image (abnormal state). Figure 2 (C) is a monitoring screen in a state where the camera image is excluded from the target of the abnormality detection process (exclusion state).

As shown in Figure 2 (A), when no abnormality is detected in the target area in the camera image, a frame line (frame image) representing the target area is drawn in black.

On the other hand, as shown in FIG. 2(B), when an abnormality is detected in the target area in the camera image, the border of the target area is highlighted; specifically, the border of the target area changes from black to red. In addition, the border is drawn thicker for a predetermined time from the time when an alarm is issued, i.e., when the message of abnormality detection and the control information (alert information) for the warning light 5 are output. Note that in the example shown in FIG. 2(B), an abnormality is detected because an obstacle (falling rocks, etc.) is present on the track within the target area.

As shown in FIG. 2(C), when a camera image is excluded from the anomaly detection process, the frame representing the target area changes from black to gray. Note that in the example shown in FIG. 2(C), a train passing through the target area is an excluded event, so the camera image is excluded from the anomaly detection process.

In this manner, in this embodiment, the frame line representing the target area is drawn in a different color (black, red, gray) depending on the detection result of the anomaly detection process and the judgment result of the exclusion judgment process. This allows the monitor to immediately grasp the detection result of the anomaly detection and the judgment result of the exclusion judgment.

In this embodiment, the color of the border is changed as an example of changing the display form of the border (frame image), but other display forms, such as the thickness or type (dotted line, etc.) of the border, may also be changed, or the border may be displayed in a flashing manner.

Furthermore, the display information showing the detection results of the anomaly detection process and the judgment results of the exclusion judgment process is not limited to a frame line (frame image) as in this embodiment. In other words, the detection results of the anomaly detection process and the judgment results of the exclusion judgment process may be shown by other forms of display information, such as a display frame of a camera image or a balloon (speech bubble).

In the example shown in FIG. 2(B), the presence of an obstacle (such as a fallen rock) on the track is detected as an abnormality around the track, but other equipment such as overhead lines may also be detected in the abnormality detection process.

Next, the schematic configuration of the monitoring server 2 will be described. FIG. 3 is a block diagram showing the schematic configuration of the monitoring server 2. FIG. 4 is an explanatory diagram showing an overview of the processing performed by the monitoring server 2. FIG. 5 is an explanatory diagram showing the timing of the display processing and the notification processing according to the detection result of the anomaly detection processing and the judgment result of the exclusion judgment processing performed by the monitoring server 2.

As shown in FIG. 3, the monitoring server 2 includes a communication unit 21, a display unit 22, an operation input unit 23, a memory 24, and a processor 25.

The communication unit 21 communicates with the camera 1, recorder 3, monitoring terminal 4, warning light 5, and user terminal 6 via the network. Specifically, the communication unit 21 receives camera images sent from the camera 1. The communication unit 21 also sends display information of the monitoring screen to the monitoring terminal 4 and receives operation information from the monitoring terminal 4. The communication unit 21 also sends control information to the warning light 5. The communication unit 21 also sends e-mails to the user terminal 6.

The display unit 22 displays an operation screen related to the operating conditions of the device.

The operation input unit 23 allows the administrator to operate the screen displayed on the display unit 22.

The memory 24 stores programs executed by the processor 25. The memory 24 also temporarily stores camera images acquired from the camera 1.

The processor 25 performs various processes related to line monitoring by executing the programs stored in the memory 24. In this embodiment, the processor 25 performs area setting process, exclusion determination process, abnormality detection process, display process, and notification process.

In the area setting process, the processor 25 sets the target area for the anomaly detection process on the camera image based on the administrator's operational input. In this embodiment, the range of the target area is set by the administrator performing an operation to specify the range of the target area on the area setting screen (see FIG. 6).

In the exclusion determination process, the processor 25 determines whether or not to exclude the real-time camera image from the target of the anomaly detection process. Specifically, the processor 25 detects the movement of the subject by comparing the camera images (frames) output sequentially from the camera 1 at each time, obtains movement information regarding the movement of the subject, and determines whether or not to exclude the camera image from the target of the anomaly detection process based on the movement information.

In the anomaly detection process, the processor 25 detects anomalies in the target area based on real-time camera images. Specifically, an anomaly in the target area is detected using a classifier (machine learning model) generated by machine learning (e.g., deep learning). This anomaly detection process makes it possible to detect anomalies around the track, i.e., a state in which safe train passage is hindered by obstacles on the track caused by falling rocks, landslides, avalanches, etc.

In the display process, the processor 25 generates a surveillance camera image (display information) by combining the camera image with a frame line (frame image) of the target area, and outputs the surveillance camera image to the recorder 3 (see FIG. 4). This allows the surveillance camera image to be displayed by the monitoring terminal 4 or the user terminal 6 by accessing the recorder 3. In this display process, the frame line of the target area is drawn in a display form (black, red, gray) according to the detection result of the anomaly detection process and the judgment result of the exclusion judgment process.

In this embodiment, in a normal state where the exclusion determination process determines not to exclude the camera image from the anomaly detection process and no abnormality is detected in the target area in the anomaly detection process, the border of the target area is drawn in black (see Figures 2(A), 5(A), and (B)). On the other hand, when the anomaly detection process detects an abnormality in the target area and transitions to an abnormal state, the border of the target area changes to red (see Figures 2(B) and 5(A)). Also, when the exclusion determination process determines to exclude the camera image from the anomaly detection process and transitions to an excluded state, the border of the target area changes to gray (see Figures 2(C) and 5(B)).

In the notification process, when the processor 25 detects an abnormality in the target area during the anomaly detection process, it generates and outputs notification information to notify the observer that an abnormality has been detected. This notification information is output when a predetermined notification condition is met. In this embodiment, the notification condition is that the notification information is output when a state in which an abnormality has been detected in sequentially input camera images (abnormal state) continues for a predetermined period of time or more. The notification information includes the time of notification, identification information of the camera 1, etc.

In this embodiment, an email containing a message about the abnormality detection is sent to the user terminal 6 as notification information for notifying the supervisor that an abnormality has been detected (see Figs. 4 and 5(A)). In addition, as the notification information, control information is sent to the warning light 5 to instruct the warning light 5 to perform a notification operation (lighting or blinking) (see Figs. 4 and 5(A)).

In addition, in the notification process, the processor 25 determines in the exclusion determination process to exclude the camera image from the anomaly detection process, and if this state continues for a predetermined time or longer, an email containing a message about the continuous exclusion state is sent to the user terminal 6 as notification information to notify the monitor that the exclusion state, i.e., the state in which anomaly detection is not being performed, has continued for a long time (see Figures 4 and 5 (B)).

In addition, in this embodiment, after transitioning to the continuous exclusion state, if the exclusion determination process determines not to exclude the camera image from the anomaly detection process, an email containing a message that the continuous exclusion state has been released is sent to the user terminal 6 as notification information to notify the monitor that the continuous exclusion state has been released, i.e., that the state has returned to one in which anomaly detection can be performed (see Figures 4 and 5 (B)).

In this manner, in this embodiment, if the state in which the camera image is excluded from the anomaly detection process (exclusion state) continues for a predetermined time or more, the monitor is notified. This allows the monitor to understand the exclusion state, i.e., that the anomaly detection process is not being carried out. Furthermore, in the case of an excluded event that continues for a long time, such as rainfall or snowfall, the monitor is notified, but in the case of an excluded event that does not require an alert and is resolved in a short time, such as the passing of a train, the monitor is not notified, and unnecessary alerts can be suppressed.

In this embodiment, the monitoring server 2 generates a surveillance camera image in which the frame line of the target area is superimposed on the camera image and transmits it to the recorder 3, but the camera image, position information of the target area on the camera image, and information on the display form (color, etc.) of the frame line of the target area may be transmitted to the recorder 3 as display information. In this case, the surveillance camera image is displayed by performing a process in the monitoring terminal 4 or the user terminal 6 to superimpose the frame line of the target area on the camera image based on the display information stored in the recorder 3.

Next, we will explain the area setting screen displayed on the display unit 22 of the monitoring server 2. Figure 6 is an explanatory diagram showing the area setting screen.

In the monitoring server 2, an operation screen related to the operating conditions of the device itself is displayed on the display unit 22. In this embodiment, an area setting screen for setting a target area for anomaly detection processing is displayed on the display unit 22 as one of the operation screens.

On this area setting screen, camera images of the monitoring area captured by camera 1 are displayed, and the administrator can specify the range of the target area on the camera image. In the example shown in Figure 6, the range of the target area is specified by inputting the positions of the vertices of a polygon representing the range of the target area. This causes a frame line (frame image) representing the range of the target area to be superimposed on the camera image. Note that if camera 1 captures images of multiple monitoring areas using the PTZ function, a target area for anomaly detection processing can be set for each monitoring area after the PTZ operation.

The anomaly detection process detects abnormalities around the track, i.e., obstacles on the track caused by falling rocks, landslides, avalanches, etc. that impede the safe passage of trains, so the target area for the anomaly detection process is set to surround a required range of the track (rails).

In this embodiment, the administrator directly operates the monitoring server 2 to specify the range of the target area, but a management terminal (not shown) may access the monitoring server 2, and the administrator may operate the management terminal to specify the range of the target area.

Next, we will explain the exclusion determination process performed by the monitoring server 2. Figure 7 is an explanatory diagram showing an overview of the exclusion determination process.

The monitoring server 2 performs an exclusion determination process that uses image analysis of the real-time camera image sent from the camera 1 to determine whether or not to exclude the real-time camera image from the anomaly detection process. In this exclusion determination process, the real-time camera image is compared with the immediately preceding camera image to detect the movement of the subject, motion information regarding the subject's motion is obtained, and based on the motion information, it is determined whether or not to exclude the real-time camera image from the anomaly detection process. Note that a method using block matching, for example, can be used for this exclusion determination process. Furthermore, in the exclusion determination process, a determination can be made based on the direction and magnitude of the motion (vector) obtained as the motion information.

By using this type of exclusion determination process, if an event that should be excluded from the anomaly detection process (an excluded event) occurs in a real-time camera image as an event that may cause a false positive in the anomaly detection process, the camera image can be excluded from the anomaly detection process.

Here, camera images in which events to be excluded, such as those listed below, occur can be excluded from the anomaly detection process by the exclusion determination process. Note that in this embodiment, the events to be excluded are not identified, but it is also possible to identify the type of each event by using a classifier or the like equipped with learning data for each event.

The first example of an event to be excluded is when the camera 1 is rotating or performing PTZ operations (pan, tilt, zoom) as shown in FIG. 7(A). The camera 1 installed along a railway line may monitor both the tracks on the uphill side and the downhill side of the installation point, in which case the camera 1 automatically rotates periodically to switch the monitoring area. Also, an observer may manually operate the camera 1 to check the situation in an area different from the normal monitoring area, in which case the camera 1 performs PTZ operations in response to the observer's operation. In the camera image while the camera 1 is rotating or performing PTZ operations, the image changes drastically overall, making it highly likely that a false detection will occur in the anomaly detection process.

The second example of an event to be excluded is when a train is passing over the track, as shown in Figure 7 (B). In this case, the train's body covers the track, so any abnormalities around the track cannot be detected. Also, since the train is passing normally, it is assumed that there are no abnormalities.

A third example of an event to be excluded is when poor image quality occurs due to heavy rainfall or snowfall, as shown in Figure 7 (C). When there is significant rainfall or snowfall, the camera image is covered with a large number of raindrops or snowflakes, making the condition around the orbit in the camera image unclear and making it difficult to properly detect anomalies.

A fourth example of an event to be excluded is when poor imaging occurs due to light from train or automobile headlights, as shown in Figure 7 (D). Camera 1 is illuminated by the light from the headlights of a train traveling on the track, and also by the light from the headlights of automobiles traveling along the road. When light from such train or automobile headlights enters camera 1, noticeable whiteout occurs in the camera image due to the movement of the train or automobile, making it difficult to properly detect abnormalities around the track in the camera image. Note that if a train or automobile is stopped, this is not an event to be excluded, and abnormality detection processing is performed.

A fifth example of an event to be excluded is when poor imaging occurs due to an object such as an insect floating near the lens of camera 1, as shown in Figure 7 (E). Animals such as insects may fly near the lens of camera 1, or a plastic bag blown up by the wind may pass by. In this case, these floating objects may appear large in the image, making the condition around the trajectory in the camera image unclear, making it difficult to properly detect anomalies. Note that a state in which a floating object such as an insect adheres to the lens of camera 1 is not an event to be excluded, and anomaly detection processing is performed.

In this embodiment, the exclusion determination process is performed based on the camera images, but it is also possible to obtain operation information related to the rotation or PTZ movement of the camera 1 from the camera 1 and perform the exclusion determination process based on the operation information. This makes it possible to exclude camera images captured during the rotation or PTZ movement of the camera 1 from the targets of the anomaly detection process (see FIG. 7(A)).

These excluded events occur temporarily and are resolved naturally. On the other hand, abnormalities around the track, i.e., obstacles on the track such as falling rocks, mudslides, and avalanches that impede the safe passage of trains, are not temporary; once they occur, they will continue until workers complete restoration work.

Next, we will explain the anomaly detection process performed by the monitoring server 2. Figure 8 is an explanatory diagram showing an overview of the anomaly detection process. Figure 8 (A) shows the situation during learning, and Figure 8 (B) shows the situation during operation.

As shown in FIG. 8(B), the monitoring server 2 performs an anomaly detection process to detect anomalies around the track in the target area based on the input real-time camera images. In this embodiment, the anomaly detection process is performed using a classifier (machine learning model) generated by machine learning (e.g., deep learning). This anomaly detection process makes it possible to detect anomalies around the track, i.e., a state in which safe train passage is hindered due to obstacles on the track caused by falling rocks, soil inflows, avalanches, etc.

Here, as shown in FIG. 8(A), the classifier is trained in advance using camera images in an abnormal state and camera images in a normal state. As a result, as shown in FIG. 8(B), when a real-time camera image is input to the classifier, the classifier outputs a detection result indicating whether or not an abnormality has been detected in the real-time camera image. Note that since it is difficult to obtain a camera image in an actual abnormal state, a pseudo camera image in an abnormal state may be generated from a camera image in a normal state.

Next, the operation procedure of the monitoring server 2 will be described. FIG. 9 is a flow diagram showing the operation procedure of the monitoring server 2. Note that the camera images (frames) at each time are input sequentially from the camera 1 to the monitoring server 2.

In the monitoring server 2, the processor 25 first acquires real-time camera images input from the camera 1 (ST101). Next, the processor 25 performs an exclusion determination process to determine whether or not to exclude the input real-time camera images from the target of the anomaly detection process (ST102).

Here, if the real-time camera images are not excluded from the anomaly detection process (No in ST103), the processor 25 then performs anomaly detection process to detect anomalies in the target area based on the real-time camera images (ST104).

If an abnormality is detected in the target area (Yes in ST105), the processor 25 then determines whether the state in which the abnormality was detected (abnormal state) has continued for a predetermined period of time or longer (ST106).

Here, if the abnormal state continues for a predetermined time or longer (Yes in ST106), the processor 25 sends an email including a message of abnormality detection to the user terminal 6 as notification information to notify the supervisor that an abnormality has been detected (ST107). The processor 25 also sends control information (notification information) to the warning light 5 as notification information instructing the warning light 5 to perform a notification operation (ST108).

Next, processor 25 determines whether the input of camera images from camera 1 has ended (ST115). If the input of camera images from camera 1 has not ended (No in ST115), the process returns to ST101 and proceeds to processing the next camera image (frame).

If the real-time camera image is to be excluded from the anomaly detection process (Yes in ST103), the processor 25 then determines whether the state in which the camera image is excluded from the anomaly detection process (exclusion state) has continued for a predetermined period of time or more (ST109).

Here, if the excluded state continues for a predetermined time or more (Yes in ST109), the processor 25 sets the state to a continuous excluded state (ST110). Next, the processor 25 sends an email to the user terminal 6 including a message about the continuous excluded state, that is, a state in which anomaly detection has not been performed has continued for a long time, as notification information to notify the monitor that the excluded state has continued for a long time (ST111). Then, proceed to ST115.

Also, if no abnormality is detected in the target area (No in ST105), the processor 25 then determines whether or not a continuous exclusion state is set (ST112).

If the continuous exclusion state is set (Yes in ST112), the processor 25 releases the continuous exclusion state (ST113). Next, the processor 25 sends an email to the user terminal 6 as notification information to notify the monitor that the continuous exclusion state has been released, including a continuous exclusion state release message indicating that the continuous exclusion state has been released, i.e., that the state has returned to one in which anomaly detection processing can be performed (ST114). Then, proceed to ST115.

In this embodiment, if the excluded state continues for a predetermined time or longer, an email containing a continuous excluded state message indicating that the excluded state has continued for a long time is sent to the user terminal 6 (ST111). At this time, the frame line of the target area in the surveillance camera image displayed on the surveillance terminal 4 may be drawn in a display form (e.g., a thick gray line) different from the display form of the excluded state (gray line).

In addition, it is preferable that the administrator be able to specify an appropriate time (e.g., a few minutes, a few tens of minutes, an hour, etc.) on the operation screen of the monitoring server 2 as the threshold time used to determine whether the excluded state has continued for a predetermined time or more (ST109).

As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to this, and can also be applied to embodiments in which modifications, substitutions, additions, omissions, etc. have been made. It is also possible to combine the components described in the above embodiments to create new embodiments.

The wayside monitoring device and wayside monitoring system of the present invention have the effect of suppressing erroneous detection in the anomaly detection process by excluding camera images that are highly likely to be erroneous in the anomaly detection process from the target of the anomaly detection process in advance, and are useful as a wayside monitoring device and wayside monitoring system that executes a process to detect abnormalities around the track based on camera images output from cameras installed along the railway line using a processor.

1 Camera 2 Surveillance server (railway monitoring device, information processing device)
3 Recorder 4 Monitoring terminal 5 Warning light 6 User terminal 21 Communication unit 22 Display unit 23 Operation input unit 24 Memory 25 Processor

Claims (9)

A wayside monitoring device that executes an abnormality detection process by a processor to detect an abnormality around a track based on camera images taken by cameras installed along a railway line,
The processor,
setting a target area of the anomaly detection process on the camera image based on an operational input by a user;
performing the anomaly detection process to detect an anomaly in the target area based on the real-time camera image output from the camera;
Furthermore, prior to the abnormality detection process,
performing an exclusion determination process for determining whether or not to exclude the real-time camera image from the target of the anomaly detection process;
When an abnormality is detected in the target area, notification information is output to notify an observer of the occurrence of the abnormality.
a monitoring camera image superimposed on the camera image and showing either a normal state, an abnormal state, or an excluded state as the state of the target area, depending on the detection result of the abnormality detection process and the judgment result of the exclusion judgment process, is generated and output . The processor,
2. The wayside monitoring device according to claim 1, wherein the abnormality detection process detects an abnormality in the target area using a classifier that has been trained in advance using the camera images in an abnormal state and the camera images in a normal state. The processor,
The trackside monitoring device according to claim 1, characterized in that, as the exclusion determination process, movement information of a subject is obtained based on the camera image, and based on the movement information, it is determined whether or not the camera image should be excluded from the anomaly detection process. The processor,
The trackside monitoring device according to claim 1, characterized in that, when an exclusion state in which a determination result that the camera image is excluded from the anomaly detection processing has been obtained continues for a predetermined time or more, a continuous exclusion state is set and the notification information indicating that is output. The processor,
The trackside monitoring device according to claim 4, characterized in that when a determination result is obtained that the camera image should not be excluded from the anomaly detection processing and the continuous exclusion state is released, the notification information indicating that fact is output. A wayside monitoring device that executes an anomaly detection process to detect anomalies around a track based on camera images taken by cameras installed along a railway line, the device comprising:
The processor,
setting a target area of the anomaly detection process on the camera image based on an operational input by a user;
performing the anomaly detection process to detect an anomaly in the target area based on the real-time camera image output from the camera;
Furthermore, prior to the abnormality detection process,
performing an exclusion determination process for determining whether or not to exclude the real-time camera image from the target of the anomaly detection process;
a monitoring camera image superimposed on the camera image and showing either a normal state, an abnormal state, or an excluded state as the state of the target area, depending on the detection result of the abnormality detection process and the judgment result of the exclusion judgment process, is generated and output. The processor,
The wayside monitoring device according to claim 6, characterized in that a display form of a frame image showing the target area is changed as the status image. A railway line monitoring system including a camera installed along a railway line and an information processing device that executes an abnormality detection process to detect an abnormality around a railway line based on a camera image captured by the camera,
The information processing device includes:
setting a target area of the anomaly detection process on the camera image based on an operational input by a user;
performing the anomaly detection process to detect an anomaly in the target area based on the real-time camera image output from the camera;
Furthermore, prior to the abnormality detection process,
performing an exclusion determination process for determining whether or not to exclude the real-time camera image from the target of the anomaly detection process;
When an abnormality is detected in the target area, notification information is output to notify an observer of the occurrence of the abnormality.
a monitoring camera image superimposed on the camera image and representing one of a normal state, an abnormal state, or an excluded state as the state of the target area, depending on the detection result of the abnormality detection process and the judgment result of the exclusion judgment process, is generated and output. A railway line monitoring system including a camera installed along a railway line and an information processing device that executes an abnormality detection process to detect an abnormality around a railway line based on a camera image captured by the camera,
The information processing device includes:
setting a target area of the anomaly detection process on the camera image based on an operational input by a user;
performing the anomaly detection process to detect an anomaly in the target area based on the real-time camera image output from the camera;
Furthermore, prior to the abnormality detection process,
performing an exclusion determination process for determining whether or not to exclude the real-time camera image from the target of the anomaly detection process;
a monitoring camera image superimposed on the camera image and representing one of a normal state, an abnormal state, or an excluded state as the state of the target area, depending on the detection result of the abnormality detection process and the judgment result of the exclusion judgment process, is generated and output.

Images