JP7743334B2 - River monitoring system and river monitoring method - Google Patents

Description

The present invention relates to a river monitoring system and a river monitoring method.

In recent years, river monitoring systems have been proposed that use machine learning models to detect water surfaces from river images (river footage) obtained from unmanned aerial vehicles (UAVs, drones, etc.) or satellites, and grasp the flooding situation of rivers.

Patent Document 1 discloses technology that determines whether the flow of a river shown in an image deviates from its normal state and whether there is a risk of flooding based on pre-learned data on the state of the river before it reached a flooding state.

Japanese Patent Application Laid-Open No. 2021-170204

By the way, machine learning models used for river monitoring are usually created by training them on many river images (river videos). In other words, the accuracy of the machine learning model improves dramatically when it is similar to the environment in which it will actually be used.

Machine learning models used in river monitoring are expected to accurately identify flooded areas. However, it is difficult to collect images of rivers that actually flood, and machine learning models that have been trained on many images (river videos) of rivers that do not flood are unable to accurately identify flooded areas when rivers flood.

The present invention was made in consideration of the above, and aims to accurately determine flood locations when rivers flood.

To solve the above-mentioned problems and achieve the objectives, the present invention comprises a training image creation unit that creates pseudo-flood images that resemble images of river flooding based on pseudo-flood labels that indicate areas predicted to be flooded in the event of a river flood and photographed images; a learning unit that associates the pseudo-flood images created by the training image creation unit with the pseudo-flood labels as training data and retrains a machine learning model for water surface detection; and a river flood detection unit that infers locations where water surfaces exist from photographed images and detects the presence or absence of river flooding based on the machine learning model retrained by the learning unit.

The present invention has the effect of enabling accurate determination of flood locations when rivers flood.

FIG. 1 is a block diagram showing a configuration of a river monitoring system according to an embodiment. FIG. 2 is a diagram showing the hardware configuration of a computer (server). FIG. 3 is a block diagram illustrating an example of the functional configuration of the river monitoring system. FIG. 4 is a diagram showing an example of a captured image and a normal label in a normal state. FIG. 5 is a diagram showing an example of a pseudo-flood image and a pseudo-flood label. FIG. 6 is a flowchart showing the flow of river monitoring processing in the river monitoring system.

Embodiments of the river monitoring system and river monitoring method are described in detail below with reference to the accompanying drawings.

Figure 1 is a block diagram showing the configuration of a river monitoring system 100 according to an embodiment. As shown in Figure 1, in the river monitoring system 100, a captured image accumulation server 10, a label storage server 20, a learning image creation server 30, a learning server 40, a river flood detection device 50, and at least one river monitoring camera 60 are connected via a network 70. The network 70 may be, for example, a LAN (Local Area Network), Ethernet (registered trademark), or the Internet.

Next, we will explain the hardware configuration of the computers (servers) that make up the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50.

Here, Figure 2 is a diagram showing the hardware configuration of a computer (server). As shown in Figure 2, the computers (servers) that make up the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50 each include a control unit 101 such as a CPU (Central Processing Unit) that controls the entire device, a main memory unit 102 such as a ROM (Read Only Memory) or RAM (Random Access Memory) that stores various data and programs, an auxiliary memory unit 103 such as an HDD (Hard Disk Drive) or CD (Compact Disc) drive that stores various data and programs, and a bus 104 that connects these. These can be realized using a hardware configuration that utilizes a typical computer. The CPU has a clock function that measures the current time.

The captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50 each perform unique functions due to differences in the programs stored in the auxiliary memory unit 103.

The control unit 101 is also connected via a bus 104 to a display unit 105 that displays information, an operation input unit 106 such as a keyboard or mouse that accepts user input instructions, and a communication I/F (interface) 107 that controls communication with external devices.

Next, we will explain the various functions of the river monitoring system 100 that are realized by the control units 101 of the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50, each of which executes a program stored in the auxiliary memory unit 103 in this hardware configuration.

Here, Figure 3 is a block diagram illustrating the functional configuration of the river monitoring system 100. As shown in Figure 3, the river monitoring system 100 includes a captured image storage unit 110, a label storage unit 120, a learning image creation unit 130, a learning unit 140, and a river flood detection unit 150.

The captured image storage unit 110 is realized by the control unit 101 of the captured image storage server 10 executing a program stored in the auxiliary memory unit 103. The label storage unit 120 is realized by the control unit 101 of the label storage server 20 executing a program stored in the auxiliary memory unit 103. The training image creation unit 130 is realized by the control unit 101 of the training image creation server 30 executing a program stored in the auxiliary memory unit 103. The learning unit 140 is realized by the control unit 101 of the learning server 40 executing a program stored in the auxiliary memory unit 103. The river flood detection unit 150 is realized by the control unit 101 of the river flood detection device 50 executing a program stored in the auxiliary memory unit 103.

The captured image storage unit 110 of the captured image storage server 10 stores the video (captured images) acquired from the river monitoring camera 60. The captured images may be arbitrarily determined by the user and stored in the captured image storage unit 110 of the captured image storage server 10, or they may be automatically stored in the captured image storage unit 110 of the captured image storage server 10 when certain conditions, such as the time of day, weather, or date, are met.

The label storage unit 120 of the label storage server 20 generates and stores normal labels by extracting the water surface area for each captured image stored in the captured image storage unit 110 of the captured image storage server 10. It also predicts areas that will be flooded in the event of a river flood and generates and stores pseudo-flood labels by extracting the water surface area. Note that normal labels and pseudo-flood labels are generated automatically or manually by the label storage unit 120 of the label storage server 20.

Figure 4 shows an example of a normal image and normal label. Figure 4(a) shows an image captured by the river monitoring camera 60, and Figure 4(b) shows a normal label for the image shown in Figure 4(a). As shown in Figure 4(b), the normal label is obtained by extracting the river area of the image captured by the river monitoring camera 60 as a water surface area.

Figure 5 also shows an example of a pseudo-flood image and pseudo-flood labels created by the learning image creation server 30, which will be described later. Figure 5(a) shows a pseudo-flood image created by the learning image creation server 30, which will be described later, and Figure 5(b) shows a pseudo-flood label for the pseudo-flood image shown in Figure 5(a). As shown in Figure 5(b), the pseudo-flood label is an extracted water surface area where flooding is predicted in the event of a river flood. In the normal label shown in Figure 4(b) and the pseudo-flood label shown in Figure 5(b), the water surface area is represented by filling in white and the rest with black.

Pseudo-flood labels are labels that annotate areas where rivers are likely to overflow during heavy rain or typhoons, even though no actual flooding has occurred, and river areas as water surface areas. Flooding locations are predicted using past data or by simulating topographical data. One method for assigning pseudo-flood labels is to use a GIS (Geographic Information System) to estimate the approximate location of a photograph from a captured image. Then, based on the estimated location, pseudo-flood labels are uniformly assigned to areas that are lower than a predetermined elevation by more than a certain height from a pre-set point. Note that multiple pseudo-flood labels may be generated depending on the cause of flooding (typhoons, heavy rain, dam discharge, etc.) and its scale, in addition to topographical data.

The training image creation unit 130 of the training image creation server 30 receives captured images from the captured image storage unit 110 of the captured image storage server 10. The training image creation unit 130 of the training image creation server 30 also receives pseudo-flood labels from the label storage unit 120 of the label storage server 20. The training image creation unit 130 of the training image creation server 30 then creates pseudo-flood images for use in machine learning that resemble images of river flooding, from the captured images received from the captured image storage unit 110 of the captured image storage server 10 and the pseudo-flood labels for positions corresponding to the captured images received from the label storage unit 120 of the label storage server 20.

More specifically, the learning image creation unit 130 of the learning image creation server 30 creates a pseudo-flood image by performing color conversion on the white areas of the pseudo-flood labels in the photographed images stored in the photographed image storage unit 110 of the photographed image storage server 10 (for example, by adding a color that imitates flooding to a photographed image of a normal river).

Figure 5(a) shows an example of a pseudo-flood image. As shown in Figure 5(a), the pseudo-flood image is created by adding colors that mimic flooding to a photographed image of a normal river.

The training image creation unit 130 of the training image creation server 30 defines the created pseudo-flood image as a "teaching image" and the pseudo-flood label used to create the pseudo-flood image as a "correct label," and associates these to create teaching data.

Then, the training image creation unit 130 of the training image creation server 30 sends the created training data to the learning unit 140 of the learning server 40, causing additional training of the water surface detection model. This training can improve the robustness of the river flood detection unit 150 of the river flood detection device 50.

The learning unit 140 of the learning server 40 retrains the water surface detection model using the pseudo-flood images and pseudo-flood labels, which are the training data received from the learning image creation unit 130 of the learning image creation server 30. In addition, the learning image creation unit 130 of the learning server 40 replaces the retrained water surface detection model with the water surface detection model of the currently operating river flood detection device 50, thereby completing the retraining phase, which will be described later.

The river flood detection unit 150 of the river flood detection device 50 determines the location of river flooding using video (photographed images) acquired from the river monitoring camera 60 and a water surface detection model trained through deep learning.

Deep learning is a type of machine learning method that uses a multi-layered neural network (deep neural network) that mimics the neurons in the human brain, and is a method that can automatically learn the characteristics of data. In this embodiment, it is supervised deep learning that uses training data. Training data is a pair of a "training image" and a "correct label" that represents the correct classification. In this embodiment, the photographed image and the pseudo-flood image are considered "training images," and the normal label and pseudo-flood label indicating the water surface area of each image are considered "correct labels."

In its initial state, the river flood detection unit 150 of the river flood detection device 50 acquires video (photographed images) of normal river flow captured by the river monitoring camera 60 as a "teacher image." Then, based on a water surface detection model (machine learning model) that uses deep learning to learn the characteristics of river flow based on the captured images (teacher images), it infers where the water surface exists from the video (photographed images) captured by the river monitoring camera 60 and detects whether or not a river is flooding.

Next, we will explain the river monitoring process in the river monitoring system 100.

Here, Figure 6 is a flowchart showing the flow of river monitoring processing in the river monitoring system 100. As shown in Figure 6, the river monitoring system 100 asynchronously operates an inference phase and a relearning phase when executing river monitoring processing. When the relearning phase ends, the river monitoring system 100 replaces the water surface detection model of the river flood detection unit 150 of the river flood detection device 50 that is currently operating with the water surface detection model after relearning.

First, let's explain the re-learning phase.

First, the captured image storage unit 110 of the captured image storage server 10 stores the video (captured images) acquired from the river monitoring camera 60 (step S1).

Next, the label storage unit 120 of the label storage server 20 generates and stores normal labels and pseudo-flood labels for the river images to be assigned to the photographed images stored in the photographed image storage unit 110 of the photographed image storage server 10 (step S2).

Next, the training image creation unit 130 of the training image creation server 30 creates pseudo-flood images to be used for training from the captured images received from the captured image storage unit 110 of the captured image storage server 10 and the pseudo-flood labels received from the label storage unit 120 of the label storage server 20 (step S3). More specifically, the training image creation unit 130 of the training image creation server 30 creates pseudo-flood images by performing color conversion (for example, applying a color that imitates the flooded river portion to a captured image of a normal river) on the areas corresponding to the water surface areas indicated by the pseudo-flood labels (white areas shown in Figure 6(b)) for the captured images stored in the captured image storage unit 110 of the captured image storage server 10.

The specific method for color conversion on the pseudo-flood labels uses the Lab color space. Specifically, for each pixel in the image at coordinates (x, y), only the part corresponding to the water surface area indicated by the pseudo-flood label is color converted using the following conversion formula:

The parameters α and β in the above conversion formula are parameters that determine the color of the assumed river when it floods.

The learning image creation unit 130 of the learning image creation server 30 regards the created pseudo-flood image as a "teaching image" and the pseudo-flood label used to create the pseudo-flood image as a "correct label," and transmits the teaching data associating these to the learning unit 140 of the learning server 40.

Next, the learning unit 140 of the learning server 40 receives training data from the learning image creation unit 130 of the learning image creation server 30 and re-learns the water surface detection model (step S4).

In addition, the learning unit 140 of the learning server 40 ends the re-learning phase by replacing the re-learned water surface detection model with the water surface detection model of the river flood detection unit 150 of the currently operating river flood detection device 50 (step S5).

In this embodiment, the label storage unit 120 of the label storage server 20 generates a normal label and a pseudo-flood label for each captured image stored in the captured image storage unit 110 of the captured image storage server 10, but this is not limited to this. For example, if the shooting direction of the river monitoring camera 60 is fixed, the label storage unit 120 of the label storage server 20 may estimate the location of the river monitoring camera 60's image capture and the locations within the location that are likely to be flooded due to rising river levels, etc., when the river monitoring camera 60 is installed, and generate and store normal labels and pseudo-flood labels. This eliminates the need to generate normal labels and pseudo-flood labels for each captured image.

In addition, the captured image storage unit 110 of the captured image storage server 10 may be configured to automatically store footage (captured images) acquired from the river monitoring camera 60 when certain conditions, such as the time of day, weather, date, etc., are met. This makes it possible to acquire captured images under different shooting conditions, and to create a large amount of training data under different conditions.

Next, we'll explain the inference phase.

At a predetermined timing, the river flood detection unit 150 of the river flood detection device 50 detects locations where a water surface exists from the video (captured image) captured by the river monitoring camera 60 based on the water surface detection model (step S11).

In this way, according to this embodiment, by detecting areas where water surfaces exist in captured images based on a water surface detection model (machine learning model) created by learning pseudo-flood images that resemble images of river flooding, the accuracy of the machine learning model used for river monitoring is improved, making it possible to accurately determine flood locations when rivers flood.

In addition, the programs executed by the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50 in this embodiment are provided as installable or executable files recorded on computer-readable recording media such as CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disc).

In addition, the programs executed by the captured image storage server 10, label storage server 20, training image creation server 30, learning server 40, and river flood detection device 50 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.In addition, the programs executed by the captured image storage server 10, label storage server 20, training image creation server 30, learning server 40, and river flood detection device 50 of this embodiment may be provided or distributed via a network such as the Internet.

Furthermore, the river monitoring system 100 of this embodiment is equipped with a captured image storage server 10, a label storage server 20, a training image creation server 30, a learning server 40, and a river flood detection device 50, but is not limited to this. All of the functions of the captured image storage unit 110, label storage unit 120, training image creation unit 130, learning unit 140, and river flood detection unit 150 may be realized by a single device, or several functions may be combined and realized by a single device.

100 River monitoring system 110 Captured image storage unit 120 Label storage unit 130 Learning image creation unit 140 Learning unit 150 River flood detection unit

Claims (4)

a learning image creation unit that creates pseudo-flood images that resemble images of river floods based on pseudo-flood labels that indicate areas that are predicted to be flooded when a river floods and on photographed images;
a learning unit that associates the pseudo-flood images created by the learning image creation unit with pseudo-flood labels to use them as training data and re-trains a machine learning model for water surface detection;
a river flood detection unit that infers locations where water surfaces exist from photographed images based on the machine learning model retrained by the learning unit and detects whether or not a river is flooding;
A river monitoring system comprising: the learning image creation unit performs color conversion on areas in the captured images where flooding is predicted and which are indicated by the pseudo-flood labels;
2. The river monitoring system according to claim 1. the learning image creation unit performs color conversion on the captured image to a color that imitates a flooded river portion;
3. The river monitoring system according to claim 2. A river monitoring method in a river monitoring system, comprising:
a learning image creation process for creating pseudo-flood images that resemble images of river floods based on pseudo-flood labels that indicate areas that are predicted to be flooded when a river floods and on photographed images;
a learning process in which the pseudo-flood images created in the learning image creation process are associated with pseudo-flood labels to be used as training data, and a machine learning model for water surface detection is retrained;
a river flood detection step of inferring locations where water surfaces exist from photographed images based on the machine learning model retrained in the learning step and detecting whether or not a river is flooding;
A river monitoring method comprising: