JP7712151B2 - Skin loss prediction device, machine learning device, skin loss prediction method, and machine learning method - Google Patents

Description

The present invention relates to a skin-fall prediction device that predicts the possibility of skin-fall at a tunnel face, a machine learning device that generates a learning model used in the skin-fall prediction device, a skin-fall prediction method that predicts the possibility of skin-fall at a tunnel face, and a machine learning method that generates a learning model that is used in the skin-fall prediction method.

At the very tip of the excavation (face) in mountain tunnel construction, it is essential to prevent serious accidents caused by falling rocks (skin falls) as much as possible. For this reason, the face is observed, locations where spalling is predicted are estimated, and measures are taken such as knocking off the locations in advance or spraying concrete at those locations. Face observation relies on the visual judgment of experienced engineers, so oversights and individual differences are likely to occur. Therefore, Non-Patent Document 1 proposes a technology that uses artificial intelligence to predict skin falls at the face.

Nishizawa et al., "Prediction of Tunnel Face Collapse Using Multimodal Deep Learning," Proceedings of the 33rd Annual Conference of the Japanese Society for Artificial Intelligence, 2019

In the conventional technology described in Non-Patent Document 1, features of the face image, weathering deterioration degree value, and crack intersection density value are extracted using Resnet 50, and a neural network is used to detect whether or not there is a possibility of skin shedding, but there was an issue that it was not possible to predict locations where there was a risk of skin shedding.

In addition, the conventional technology evaluates the possibility of skin shedding only from the weathering deterioration degree value and the crack intersection density value in addition to the face image, which has the problem of low accuracy and versatility. Furthermore, there is also the problem that the factors for skin shedding prediction become a black box when only extracting features with Resnet50 as in the conventional technology.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide a skin-fall prediction device capable of predicting positions at risk of skin-fall, a machine learning device that generates a learning model to be used in the skin-fall prediction device, a skin-fall prediction method capable of predicting positions at risk of skin-fall, and a machine learning method that generates a learning model to be used in the skin-fall prediction method.

In order to achieve the above object, the skin fall prediction device of the present invention is a skin fall prediction device that predicts skin falls at a tunnel face, and includes a judgment data acquisition unit that acquires judgment data including a prediction image in which the tunnel face is captured, a learned model storage unit that stores a learning model that has been machine-learned to learn the correlation between input data including a learning image in which the tunnel face is captured and output data including data related to the skin fall prediction indication points and the skin fall factor pattern indication points contained in the learning image, and an inference unit that inputs the judgment data acquired by the judgment data acquisition unit into the learning model and infers the skin fall prediction points of the tunnel face captured in the prediction image , and the prediction image is cut out into a plurality of image data and used, each of the plurality of image data has an overlapping portion with other image data, each of the plurality of image data has a weighting coefficient, and the weighting coefficient of the image data is input to the learning model together with the image data, the overlapping portion includes a plurality of images divided into a grid shape, and the weighting coefficient is higher at the peripheral portion of the tunnel face than at the central portion .

The skin loss prediction device according to the present invention is also characterized in that the predicted skin loss areas are marked in the prediction image.

The skin loss prediction device according to the present invention is also characterized in that the prediction basis for the predicted skin loss areas is visualized in the prediction image.

Further, the machine learning device according to the present invention is a machine learning device that generates a learning model used in a skin fall prediction device that predicts skin fall at a tunnel face, and includes a learning data storage unit that stores multiple sets of learning data consisting of input data including a learning image in which the tunnel face is captured, and output data including data related to skin fall prediction indication points and skin fall factor pattern indication points included in the learning image, a machine learning unit that inputs multiple sets of the learning data to the learning model to machine learn the correlation between the input data and the output data, and a learned model storage unit that stores the learning model machine-learned by the machine learning unit , and the learning image is cut out into multiple image data and used, each of the multiple image data has an overlapping portion with other image data, each of the multiple image data has a weighting coefficient, and the weighting coefficient of the image data is input to the learning model together with the image data, the overlapping portion includes multiple images divided into a grid shape, and the weighting coefficient is higher at the peripheral portion than at the center portion of the tunnel face .

In addition, the skin fall prediction method of the present invention is a skin fall prediction method for predicting skin falls at a tunnel face, and includes a judgment data acquisition process for acquiring judgment data including a prediction image in which the tunnel face is captured, and an inference process for inputting the judgment data acquired by the judgment data acquisition process into a learning model that has been machine-learned to learn the correlation between input data including a learning image in which the tunnel face is captured and output data including data related to skin fall prediction indication points and skin fall factor pattern indication points contained in the learning image, and inferring the skin fall prediction points of the tunnel face captured in the prediction image , wherein the prediction image is cut out into a plurality of image data and used, each of the plurality of image data has an overlapping portion with other image data, each of the plurality of image data has a weighting coefficient, and the weighting coefficient of the image data is input to the learning model together with the image data, the overlapping portion includes a plurality of images divided into a grid shape, and the weighting coefficient is higher at the peripheral portion of the tunnel face than at the central portion .

Further, the machine learning method according to the present invention is a machine learning method for generating a learning model used in a skin fall prediction method for predicting skin fall at a tunnel face, and includes a learning data storage step for storing multiple sets of learning data in a learning data storage unit, the learning data being composed of input data including a learning image in which the tunnel face is captured, and output data including data relating to skin fall prediction indication points and skin fall factor pattern indication points included in the learning image; a machine learning step for inputting multiple sets of the learning data into the learning model, thereby making the learning model learn by machine learning the correlation between the input data and the output data; and a learned model storage step for storing the learning model machine-learned by the machine learning step in a learned model storage unit , the learning image being cut out into multiple image data, each of the multiple image data having an overlapping portion with other image data, each of the multiple image data having a weighting coefficient, the weighting coefficient of the image data being input to the learning model together with the image data, the overlapping portion including multiple images divided into a grid shape, and the weighting coefficient being higher in the peripheral portion than in the central portion of the tunnel face .

According to the skin loss prediction device of the present invention, the predicted skin loss locations are inferred based on the indicated skin loss factor patterns in addition to the indicated skin loss predicted locations, so it is possible to predict locations at risk of skin loss with greater accuracy. In addition, because inference is made based on the indicated skin loss factor patterns, it is possible to prevent skin loss prediction factors from becoming a black box.

In addition, the machine learning device according to the present invention can generate a learning model to be used in a skin loss prediction device that can predict areas at risk of skin loss with greater accuracy.

According to the skin loss prediction method of the present invention, the predicted skin loss locations are inferred based on the indicated skin loss factor patterns in addition to the indicated skin loss predicted locations, so that it is possible to predict locations at risk of skin loss with greater accuracy. In addition, because inference is made based on the indicated skin loss factor patterns, it is possible to prevent skin loss prediction factors from becoming a black box.

In addition, the machine learning method according to the present invention can generate a learning model to be used in a skin loss prediction method that can predict areas at risk of skin loss with greater accuracy.

1 is an overall configuration diagram showing an example of a skin fall prediction system 1 according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a machine learning device 4 according to an embodiment of the present invention. 10A to 10C are diagrams illustrating an example of a process for creating learning data in the skin fall prediction system 1 according to an embodiment of the present invention. This shows a classification table of the areas that indicate skin sagging causes. 1 is a diagram illustrating an example of processing a learning image 30 in which a tunnel face 10 is captured. FIG. FIG. 13 is a diagram illustrating an example of a data set of learning data. FIG. 13 is a diagram illustrating weighting coefficients used in learning model 2. 1 is a schematic diagram showing an example of a neural network model 20 applied to a learning model 2 according to an embodiment of the present invention. FIG. FIG. 2 is a diagram explaining the inference results obtained at each output layer. FIG. 2 is a block diagram showing an example of a skin fall prediction device 5 according to an embodiment of the present invention. 9 is a hardware configuration diagram showing an example of a computer 900 constituting each of the devices 3 to 6 of the skin fall prediction system 1. FIG. A flowchart showing an example of a machine learning method by the machine learning device 4 according to the embodiment of the present invention. 4 is a flowchart showing an example of a skin fall prediction method performed by the skin fall prediction device 5 according to the embodiment of the present invention. 11A and 11B are diagrams illustrating an example of an output display by the skin fall prediction device 5 according to the embodiment of the present invention.

Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. In the following, the scope necessary for the explanation to achieve the object of the present invention will be shown in a schematic manner, and the scope necessary for the explanation of the relevant part of the present invention will be mainly explained, and the parts that are omitted from the explanation will be based on the publicly known technology.
(Embodiments of the present invention)
1 is an overall configuration diagram showing an example of a skin fall prediction system 1 according to an embodiment of the present invention. The skin fall prediction system 1 is a system that obtains information about a tunnel face 10 at the forefront of excavation in mountain tunnel construction, and predicts a skin fall at the tunnel face 10 based on the information. For this purpose, the skin fall prediction system 1 has a machine learning device 4 using artificial intelligence technology, and a skin fall prediction device 5 that performs inference based on learning data learned by the machine learning device 4.

The skin fall prediction system 1 includes an imaging device 3 that images the tunnel face 10, a machine learning device 4 that operates as the subject of the learning phase of the machine learning, a skin fall prediction device 5 that operates as the subject of the inference phase of the machine learning, and a terminal device 6 used by the manager or worker of the skin fall prediction system 1. The devices 3 to 6 of the skin fall prediction system 1 are connected to each other via a network 7 so that they can communicate with each other. Note that the system configuration of the skin fall prediction system 1 shown in FIG. 1 is an example, and can be changed as appropriate. For example, in this embodiment, the skin fall prediction device 5, the machine learning device 4, and the terminal device 6 are each configured as independent information processing devices, but they can also be configured as a single information processing device.

The imaging device 3 includes a camera configured with an image sensor such as a CMOS sensor or a CCD sensor, and captures an image of the entire tunnel face 10. The imaging device 3 may be a handheld or fixed camera, and may be operated by an operator to capture an image, or may capture an image automatically when a predetermined imaging condition is met. In this embodiment, the entire tunnel face 10 is captured by one imaging device 3, but the tunnel face 10 may be captured by multiple imaging devices 3.

The imaging device 3 captures an image of the tunnel face 10 within the angle of view of the imaging device 3, and outputs the image as digital data based on a predetermined image format. As shown in FIG. 1, the imaging device 3 may be provided as an imaging device 3 connected to the machine learning device 4 and an imaging device 3 connected to the skin fall prediction device 5, or one imaging device 3 may be connected to both the machine learning device 4 and the skin fall prediction device 5 and used in common. Also, in FIG. 1, for the sake of simplicity, one imaging device 3 is connected to each of the machine learning device 4 and the skin fall prediction device 5, but multiple imaging devices 3 may be connected to each.

The machine learning device 4 is composed of a general-purpose or dedicated computer, etc., and operates as the subject of the learning phase in machine learning. The machine learning device 4 performs machine learning of the learning model 2 using learning data including images (learning images) captured by the imaging device 3. The machine learning device 4 provides the learned learning model 2 to the skin fall prediction device 5 via the network 7, a recording medium, etc. An example of the computer configuration will be described later.

The skin fall prediction device 5 is composed of a general-purpose or dedicated computer, etc., and operates as the main body of the inference phase in machine learning. The skin fall prediction device 5 predicts the tunnel face 10 from the image (prediction image) captured by the imaging device 3 using the learning model 2 that has been trained by the machine learning device 4.

The terminal device 6 is composed of a general-purpose or dedicated computer, etc. The terminal device 6 accepts various operation inputs via an input screen in order to prepare learning data, perform machine learning of the learning model 2, and predict the fall of the tunnel face 10 in the fall prediction system 1, and displays various information via a display screen such as an application or a browser. In addition, although one terminal device 6 is illustrated in FIG. 1 for simplification, a plurality of terminal devices 6 may be provided.
(Machine learning device 4)
2 is a block diagram showing an example of the machine learning device 4 according to an embodiment of the present invention. The machine learning device 4 includes a control unit 40, a communication unit 41, a training data storage unit 42, and a trained model storage unit 43.

The imaging device 3 captures a learning image 30 and transmits it to the machine learning device 4. The control unit 40 of the machine learning device 4 functions as a learning data acquisition unit 400 and a machine learning unit 401. The communication unit 41 is connected to an external device (e.g., the imaging device 3, the skin fall prediction device 5, and the terminal device 6, etc.) via the network 7, and functions as a communication interface for transmitting and receiving various data.

The learning data acquisition unit 400 is connected to an external device via the communication unit 41 and the network 7, and acquires learning data that is composed of corresponding input data and output data.

The learning data storage unit 42 is a database that stores multiple sets of learning data acquired by the learning data acquisition unit 400. The specific configuration of the database that constitutes the learning data storage unit 42 may be designed as appropriate.

The machine learning unit 401 performs machine learning using the learning data stored in the learning data storage unit 42. That is, the machine learning unit 401 generates the learning model 2 by inputting multiple sets of learning data to the learning model 2 and having the learning model 2 learn the correlation between the input data and output data that constitute the learning data.

The trained model storage unit 43 is a database that stores the trained learning model 2 trained by the machine learning unit 401. The trained model 2 stored in the trained model storage unit 43 is provided to the actual system (e.g., skin fall prediction device 5) via the network 7 or a recording medium. The trained model 2 may be provided to an external computer (e.g., a server-type computer or a cloud-type computer) and stored in the storage unit of the external computer. Also, although the trained data storage unit 42 and the trained model storage unit 43 are shown as separate storage units in FIG. 2, they may be configured as a single storage unit.

Figure 3 is a diagram illustrating an example of the process for creating learning data in the skin fall prediction system 1 according to an embodiment of the present invention. Figure 3 (A) is an image of the entire tunnel face 10 captured by the imaging device 3. Figure 3 (B) is a diagram showing the labeling of predicted skin fall areas pointed out by a skilled technician in the image, with the area surrounded by a dotted line in the figure showing the labeling of the predicted skin fall areas. This labeling becomes the learning data in the skin fall prediction system 1 according to an embodiment of the present invention.

Figure 3(C) shows the labeling of the areas where the skin-fall pattern was pointed out by the skilled technician in the image, and the area surrounded by a frame in the figure shows the labeled area where the skin-fall pattern was pointed out. Figure 4 shows a classification table of the areas where the skin-fall pattern was pointed out, and based on this classification, the skilled technician specifies areas in the learning image of the tunnel face 10 that are likely to be the cause of skin-fall, along with the classification.

In the skin fall prediction system 1 according to the embodiment of the present invention, the skin fall factor patterns are classified into eight factor patterns, which are skin fall factor patterns caused by geology: (1) traces of rock masses falling off, (2) uneven face along the cracks, (3) fine cracks, (4) clay/inclusions in the cracks, (5) weathered discoloration compared to the surrounding area, severe weathered discoloration, (6) fractured zone, fractured, sedimentation, (7) seepage, spring water, (8) others, and learning data based on these is used to prevent the factors of skin fall prediction from becoming a black box. The classification of skin fall factor patterns is not limited to these eight factor patterns and can be selected appropriately. In short, as long as at least two or more skin fall factor patterns are set, the skin fall prediction system 1 according to the present invention can be configured. FIG. 3(C) shows a case in which a skilled technician has pointed out (2) uneven face along the cracks, and (3) fine cracks as skin fall factor patterns, and these skin fall factor patterns become learning data in the skin fall prediction system 1.

The learning image 30 included in the input data is an image of the entire tunnel face 10 captured by the imaging device 3. Here, the learning image 30 capturing the entire tunnel face 10 is divided and used as data for learning, and is also divided and provided for inference as a prediction image during inference by the surface fall prediction system 1 according to the present invention.

Figure 5 is a diagram for explaining an example of processing a learning image 30 in which a tunnel face 10 has been captured. The learning image 30 or prediction image 31 in which the tunnel face 10 has been captured is divided into a grid of multiple squares after the tunnel face 10 has been recognized based on well-known image analysis techniques. From an image divided into such grids, as shown in Figure 5(A), for example, an image G1 consisting of four grids in the upper left corner is cut out, and this image data G1 is used for learning, etc. Next, as shown in Figure 5(B), image data G2 is cut out, and this image data is used for learning, etc.

As can be seen from Figures 5(A) and 5(B), image data G1 and image data G2 are cut out so that two grids (the two right grids in image data G1 and the two left grids in image data G2) overlap. Similarly, each image data is cut out successively so as to scan from the top left to the bottom right (as shown in Figures 5(C) to 5(D)), with adjacent image data above and below and to the left and right, and overlapping grids included, and used for learning, judgment, etc. In the skin loss prediction system 1 according to the present invention, image data including such overlapping grids is used for learning, judgment, etc., so that no areas are missed and risky areas for skin loss can be predicted with high accuracy.

In the example shown in FIG. 5, one piece of cut-out image data is configured with four grids, but the number of grids that the cut-out image data is configured with is arbitrary.

The data extracted from the learning images and prediction images based on the grid as described above is referred to as "image data" in this specification.

Now, the machine learning device 4 performs learning using image data (hereinafter, generalized as Gn, where n is a natural number) cut out from the learning image 30 as described above. FIG. 6 is a diagram showing an example of a dataset of learning data. Here, the input data is image data Gn, and the output data is, for example, what is called a correct answer label in supervised learning.

If the area of the predicted skin loss in the image data Gn is equal to or larger than the specified area, a correct answer label of "Predicted skin loss" is set as the output data. Also, if the area of the predicted skin loss in the image data Gn is smaller than the specified area, a correct answer label of "No predicted skin loss" is set as the output data.

In addition, a correct label is set according to the ratio of skin loss factor patterns (1) to (8) in the image data Gn. In the example of FIG. 6, the ratio (2) (e.g., area ratio) is set as the correct label as the skin loss factor pattern, and the ratio (e.g., area ratio) that does not fall into any of (1) to (8) is set as the correct label as the skin loss factor pattern. The sum of the respective correct labels based on the classification of skin loss factor patterns is 1.

In addition, if the area of the predicted skin loss in the image data Gn is equal to or larger than a predetermined area, the correct answer label "1" is set as the possibility of skin loss. In addition, if the area of the predicted skin loss in the image data Gn is smaller than a predetermined area, the correct answer label "0" is set as the possibility of skin loss.

Next, the weighting coefficients used in the skin fall prediction system 1 according to the present invention will be explained. FIG. 7 is a diagram explaining the weighting coefficients used in the learning model 2 of the skin fall prediction system 1 according to the present invention. It is empirically known that the probability of skin fall at the tunnel face 10 is higher at the periphery than at the center of the tunnel face 10, and furthermore, is higher at the upper part of the tunnel face 10 than at the lower part.

Therefore, when image data Gn is processed by learning model 2, weighting is performed using a weighting factor depending on which of the four grids the image data Gn is composed of. In the example shown in FIG. 7, three types of grids are set: a grid with a weighting factor of 1.3, a grid with a weighting factor of 1.0, and a grid with a weighting factor of 0.7. The value and type of weighting factor are arbitrary, but the weighting factor value is set so that the peripheral part of the tunnel face 10 is higher than the center, and further, the upper part of the tunnel face 10 is higher than the lower part.

For image data that is composed of grids with different weighting factors, for example, the average value of the four weighting factors can be used as the weighting factor for that image data. For example, the weighting factor for the image data Gn shown in FIG. 7 can be (1.0+1.0+1.0+1.3)/4=1.075.

Figure 8 is a schematic diagram showing an example of a neural network model 20 applied to the learning model 2 according to an embodiment of the present invention. The neural network model 20 employs a convolutional neural network (CNN) as a specific method of machine learning. The neural network model 20 includes an input layer 21, a first intermediate layer 22a, a second intermediate layer 22b, a first output layer 23a, a second output layer 23b, and a third output layer 23c.

In the neural network model 20, a first intermediate layer 22a, a first output layer 22
In the first intermediate layer 22a, data processing is performed on the predicted skin fall location, and in the second intermediate layer 22b and the second output layer 23b, data processing is performed on the skin fall cause pattern location. The output from the first output layer 23a is weighted by the weighting coefficient described above and input to the fully connected layer 222c having the third output layer 23c. The output from the second output layer 23b is also input to the fully connected layer 222c having the third output layer 23c.

The input layer 21 has neurons whose number corresponds to the number of pixels in the image data Gn as input data, and the pixel value of each pixel is input to each neuron.

The first intermediate layer 22a and the second intermediate layer 22b are respectively composed of convolutional layers 220a, 220b, pooling layers 221a, 221b, and fully connected layers 222a, 222b. The same convolutional layers and pooling layers can be used for each layer. The convolutional layers 220a, 220b and the pooling layers 221a, 221b are, for example, arranged in multiple alternating layers. The convolutional layers 220a, 220b and the pooling layers 221a, 221b extract features from the image input via the input layer 21.

Each of the fully connected layers 222a and 222b converts the two-dimensional array of features extracted from the image by the respective convolution layers 220a and 220b and pooling layers 221a and 221b, for example, using an activation function, and outputs the converted features as a one-dimensional array of feature vectors. Note that multiple fully connected layers 222a and 222b may be provided.

The output layer 23a outputs output data including the results of determining the predicted areas of skin loss included in the image data Gn based on the feature vectors output from the fully connected layer 222a. The output layer 23b outputs output data including the results of determining the areas of skin loss pattern included in the image data Gn based on the feature vectors output from the fully connected layer 222b.

Figure 9 is a diagram explaining the inference results obtained at each output layer. As shown in Figure 9, output layer 23a outputs "Skin fall predicted areas present", "Skin fall predicted areas present". In addition, output layer 23b outputs "Ratio of category (1)", "Ratio of category (2)", "Ratio of category (3)", "Ratio of category (4)", "Ratio of category (5)", "Ratio of category (6)", "Ratio of category (7)", "Ratio of category (8)", and "Ratio not falling into any of categories (1)-(8)".

The output from the first output layer 23a is weighted by a weighting coefficient and then input to the fully connected layer 222c. The output from the second output layer 23b is input to the fully connected layer 222c. The third output layer 23c is set to output a "skin drop possibility" value between 0 and 1.

Between each layer of the neural network models 20a, 20b, there are synapses that connect the neurons between the layers, and weights are assigned to each synapse in the convolutional layers 220a, 220b and the fully connected layers 222a, 222b of the intermediate layers 22a, 22b.

The machine learning unit 401 inputs the training data into the neural network model 20, and causes the neural network model 20 to machine-train the correlation between the input data (image data Gn cut out from the training image 30) and the output data. Specifically, the machine learning unit 401 inputs the image data Gn constituting the training data as input data to the input layer 21 of the neural network model 20. Note that the machine learning unit 401 may apply a predetermined image adjustment (e.g., image format, image size, image filter, image mask, etc.) to the image data Gn as pre-processing when inputting the training image 30 to the input layer 21.

The machine learning unit 401 uses an error function that compares the output data output from the output layer 23 as an inference result with the output data (correct label) that constitutes the learning data, and iteratively adjusts (backpropagation) the weights associated with each synapse so as to reduce the evaluation value of the error function.

When the machine learning unit 401 determines that a predetermined learning termination condition, such as the above series of processes being repeated a predetermined number of times or the evaluation value of the error function being smaller than a tolerance, is satisfied, the machine learning unit 401 terminates the machine learning and stores the neural network model 20 at that time (a weight parameter group consisting of all weights associated with each synapse) in the trained model storage unit 43 as the trained learning model 2. Note that when performing machine learning of the learning model 2, the machine learning unit 401 may adopt, for example, online learning, batch learning, mini-batch learning, or the like as a method for adjusting the weights, or may adopt, for example, a hold-out method, cross-validation, or the like as a method for dividing a plurality of sets of learning data into training data and test data and evaluating the learning model 2, or may determine that the misclassification rate is minimum as a predetermined learning termination condition.
(Skin fall prediction device 5)
10 is a block diagram showing an example of a skin fall prediction device 5 according to an embodiment of the present invention. The skin fall prediction device 5 includes a control unit 50, a communication unit 51, and a trained model storage unit 52.

The control unit 50 functions as a judgment data acquisition unit 500, an inference unit 501, and an output processing unit 502. The communication unit 51 is connected to an external device (e.g., the imaging device 3, the machine learning device 4, and the terminal device 6, etc.) via the network 7, and functions as a communication interface for transmitting and receiving various types of data.

The judgment data acquisition unit 500 is connected to an external device via the communication unit 51 and the network 7, and acquires judgment data including a prediction image 31 in which the tunnel face 10 is captured. The prediction image 31 is cut out as image data consisting of four grids, as in the case of learning, and used for skin fall prediction. The image data cut out from the prediction image 31 is referred to as Hn (n is a natural number). The skin fall prediction device 5 outputs a "skin fall possibility" between 0 and 1 for the image data Hn cut out from the prediction image 31. The prediction image 31 corresponds to the learning image 30 when the learning model 2 is machine-learned by the machine learning device 4. In addition, for each Hn, a weight coefficient is calculated and input to the learning model 2, as in the case of learning. The judgment data includes at least the prediction image 31, but can also include other data useful for skin fall prediction.

The inference unit 501 performs an inference process to infer the “possibility of skin falling” of the tunnel face 10 by inputting the image data Hn acquired by the judgment data acquisition unit 500 into the learning model 2.

The trained model storage unit 52 is a database that stores trained learning models 2 used in the inference process of the inference unit 501. The number of trained models 2 stored in the trained model storage unit 52 is not limited to one, and for example, multiple trained models with different machine learning methods may be stored and selectively available. The trained model storage unit 52 may also be substituted by a storage unit of an external computer (e.g., a server-type computer or a cloud-type computer), and in that case, the inference unit 501 may perform the above-mentioned inference process by accessing the external computer.

The output processing unit 502 performs output processing to output the prediction result of the "skin-fall possibility" for the image data Hn of the tunnel face 10 inferred by the inference unit 501. As the specific output means for outputting the prediction result, various means can be adopted. For example, the output processing unit 502 may transmit the judgment result to the terminal device 6, further display it on the screen, or may store the judgment result in the memory unit of the skin-fall prediction device 5.

The learning model 2 here is a learning model 2 in which the weighting parameter group has been adjusted (trained) by using multiple sets of learning data in the machine learning device 4 to machine-learn the relationship between the image data Gn cut out from the learning image 30 and the predicted skin loss indication areas and the skin loss cause pattern indication areas. Therefore, the inference unit 501 can infer the "skin loss possibility" in the image data Hn of the prediction image 31 by inputting the image data Hn cut out from the prediction image 31 and the weighting coefficient to the learning model 2.

In addition, it is preferable that the prediction results of skin fall for the image data Hn contained in the prediction image 31 are stored in the learned model memory unit 52 or another memory device (not shown), and past prediction results may be used as learning data to be used for online learning or re-learning, for example, to further improve the inference accuracy of the learned learning model 2.
(Configuration of computer 900)
FIG. 11 is a hardware configuration diagram showing an example of a computer 900 constituting each of the devices 3 to 6 of the skin fall prediction system 1. As shown in FIG.

Each of the devices 3 to 6 in the skin fall prediction system 1 is composed of a general-purpose or dedicated computer 900. The main components of the computer 900 include a bus 910, a processor 912, a memory 914, an input device 916, an output device 917, a display device 918, a storage device 920, a communication I/F (interface) unit 922, an external device I/F unit 924, an I/O (input/output) device I/F unit 926, and a media input/output unit 928. Note that the above components may be omitted as appropriate depending on the purpose for which the computer 900 is used.

The processor 912 is composed of one or more arithmetic processing devices (such as a CPU (Central Processing Unit), MPU (Micro-processing unit), DSP (Digital signal processor), GPU (Graphics Processing Unit)) and operates as a control unit that controls the entire computer 900. The memory 914 stores various data and programs 930, and is composed of, for example, a volatile memory (DRAM, SRAM, etc.) that functions as a main memory, and a non-volatile memory (ROM, flash memory, etc.).

The input device 916 is, for example, a keyboard, a mouse, a numeric keypad, an electronic pen, etc., and functions as an input unit. The output device 917 is, for example, a sound (audio) output device, a vibration device, etc., and functions as an output unit. The display device 918 is, for example, a liquid crystal display, an organic EL display, electronic paper, a projector, etc., and functions as an output unit. The input device 916 and the display device 918 may be integrated, such as a touch panel display. The storage device 920 is, for example, a hard disk drive (HDD), a solid state drive (SSD), etc., and functions as a memory unit. The storage device 920 stores various data necessary for the execution of the operating system and the program 930.

The communication I/F unit 922 is connected to a network 940 (which may be the same as the network 7 in FIG. 1) such as the Internet or an intranet by wire or wirelessly, and functions as a communication unit that transmits and receives data to and from other computers according to a predetermined communication standard. The external device I/F unit 924 is connected to an external device 950 such as a camera, printer, scanner, or reader/writer by wire or wirelessly, and functions as a communication unit that transmits and receives data to and from the external device 950 according to a predetermined communication standard. The I/O device I/F unit 926 is connected to an I/O device 960 such as various sensors and actuators, and functions as a communication unit that transmits and receives various signals and data, such as detection signals from sensors and control signals to actuators, between the I/O device 960. The media input/output unit 928 is composed of a drive device such as a DVD (Digital Versatile Disc) drive or a CD (Compact Disc) drive, and reads and writes data to media (non-transient storage media) 970 such as DVDs and CDs.

In the computer 900 having the above configuration, the processor 912 calls up the program 930 stored in the storage device 920 into the memory 914, executes it, and controls each part of the computer 900 via the bus 910. The program 930 may be stored in the memory 914 instead of the storage device 920. The program 930 may be recorded on the medium 970 in an installable file format or an executable file format, and provided to the computer 900 via the media input/output unit 928. The program 930 may be provided to the computer 900 by downloading it via the network 940 via the communication I/F unit 922. In addition, the computer 900 may realize various functions realized by the processor 912 executing the program 930 using hardware such as an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit).

The computer 900 is, for example, a stationary computer or a portable computer, and is an electronic device of any type. The computer 900 may be a client computer or an edge computer, or may be a server computer or a cloud computer, depending on the use of the computer 900.
(Machine learning methods)
FIG. 12 is a flowchart showing an example of a machine learning method performed by the machine learning device 4 according to an embodiment of the present invention.

First, in step S100, the learning data acquisition unit 400 prepares a desired number of pieces of learning data as a preliminary step for starting machine learning, and stores the prepared learning data in the learning data storage unit 42. The number of pieces of learning data to be prepared here may be set in consideration of the inference accuracy required for the learning model 2 that is ultimately obtained.

There are several methods that can be used to prepare the learning data. For example, the imaging device 3 is used to capture a learning image 30 of the tunnel face 10, and an experienced technician uses the terminal device 6 to record the predicted areas of skin fall and the areas of skin fall cause patterns in the learning image 30. By repeating this process every time a new tunnel face 10 appears, it is possible to prepare multiple sets of learning data.

Next, in step S110, the machine learning unit 401 prepares a pre-learning learning model 2 to start machine learning. The pre-learning learning model 2 prepared here employs the convolutional neural network illustrated in Fig. 8, and the weights of each synapse are set to an initial value. Each neuron in the input layer 21 is associated with each pixel of image data Gn cut out from the learning image 30 as input data constituting the learning data.
Each neuron in the first output layer 23a is associated with the presence or absence of a predicted skin loss point as output data constituting the learning data. Each neuron in the second output layer 23b is associated with the ratio of each classification based on the predicted skin loss pattern point as output data constituting the learning data. Each neuron in the third output layer 23c is associated with the possibility of skin loss as output data constituting the learning data.

Next, in step S120, the machine learning unit 401 acquires, for example, one set of learning data randomly from the multiple sets of learning data stored in the learning data storage unit 42.

Next, in step S130, the machine learning unit 401 inputs the input data included in the set of learning data to the input layer of the prepared learning model 2 before learning (or during learning). As a result, output data is output as an inference result from the output layer of the learning model 2, but this output data was generated by the learning model 2 before learning (or during learning). Therefore, in the pre-learning (or during learning) state, the output data output as an inference result indicates information different from the output data (correct answer label) included in the learning data.

Next, in step S140, the machine learning unit 401 performs machine learning by comparing the output data (correct label) included in the set of learning data acquired in step S120 with the output data output from the output layer as an inference result in step S130 and adjusting the weight of each synapse. In this way, the machine learning unit 401 causes the learning model 2 to learn the correlation between the input data and the output data.

Next, in step S150, the machine learning unit 401 determines whether a predetermined learning termination condition has been met, for example, based on the evaluation value of an error function based on the inference result and the output data (correct answer label) included in the learning data, or the remaining amount of unlearned learning data stored in the learning data storage unit 42.

If, in step S150, the machine learning unit 401 determines that the learning termination condition is not satisfied and machine learning is to be continued (No in step S150), the process returns to step S120, and steps S120 to S140 are performed multiple times on the learning model 2 being trained using untrained learning data. On the other hand, in step S150, if the machine learning unit 401 determines that the learning termination condition is satisfied and machine learning is to be terminated (Yes in step S150), the process proceeds to step S160.

In step S160, the machine learning unit 401 stores the trained learning model 2 (adjusted weight parameter group) trained by adjusting the weights associated with each synapse in the trained model storage unit 43, and ends the series of machine learning methods shown in FIG. 8. In the machine learning method, step S100 corresponds to the training data storage process, steps S110 to S150 correspond to the machine learning process, and step S160 corresponds to the trained model storage process. Note that in the series of machine learning methods shown in FIG. 12, the case where online learning is adopted as a method for adjusting the weights has been described, but batch learning or mini-batch learning (for example, 100 sets of training data units) may also be adopted. In addition, when multiple sets of training data are divided into training data and test data, the series of machine learning methods shown in FIG. 12 may be executed using the training data. Furthermore, it may be determined whether a predetermined learning end condition is met based on the misjudgment rate.

As described above, according to the machine learning device 4 and machine learning method of this embodiment, it is possible to provide a learning model 2 that can infer (judge, predict) with high accuracy a prediction of the surface fall of the tunnel face 10 from a learning image 30 in which at least a portion of the tunnel face 10 is captured by the imaging device 3.
(Skin loss prediction method)
Fig. 13 is a flow chart showing an example of a skin fall prediction method by the skin fall prediction device 5 according to an embodiment of the present invention. The series of skin fall prediction methods shown in Fig. 13 are repeatedly executed at a predetermined timing by the skin fall prediction device 5. The predetermined timing may be any timing, and may be, for example, when the imaging device 3 captures a new prediction image 31, or when a predetermined event occurs (when a judgment operation by an administrator or worker is accepted from the terminal device 6, when a judgment command is received from the skin fall prediction system 1, etc.).

In step S200, the tunnel face 10 is imaged by the imaging device 3, and the prediction image 31 is transmitted to the surface fall prediction device 5, whereby the judgment data acquisition unit 500 acquires judgment data including the prediction image 31.

Next, in step S210, a process is performed to extract multiple image data Hn from the prediction image based on the grid as described above. Then, in step S220, the weighting coefficient for each image data Hn is calculated in the manner described above.

Next, in step S230, the inference unit 501 inputs the image data Hn and its weighting coefficient to the input layer of the trained learning model 2, and obtains the "skin loss possibility" for the image data Hn as a value between 0 and 1 from the output layer of the learning model 2 as an inference result.

In step S240, it is determined whether or not an inference result has been obtained for all image data Hn. If the result of this determination is No, in step S280, the next target image data is selected, and the process proceeds to step S230 again. On the other hand, if the result of this determination is Yes, the process proceeds to step S250.

In step S250, the inference results based on the image data Hn are restored to those corresponding to the prediction image. As an inference result, a "skin removal possibility" value is obtained for each image data Hn, but since there are overlapping grids in the image data Hn, a different "skin removal possibility" value may be obtained for a certain grid. In such a case, in order to err on the safe side, it is better to assign a higher "skin removal possibility" value to that grid.

In addition, in step S250, in the prediction image in which a "skin loss possibility" value is associated with each grid as described above, a marker is generated so that areas where the skin loss possibility value is equal to or greater than a predetermined value can be displayed as "skin loss predicted areas."

In the next step S260, if there is a location in the prediction image where the skin loss possibility value is equal to or greater than a predetermined value, a process is executed to visualize the prediction basis. This provides insight into the perspectives used by the neural network to select the skin loss prediction location, and in this sense, it is possible to prevent skin loss prediction from becoming a black box. Note that the process for visualizing the prediction basis can use well-known algorithms such as Gradient-weighted Class Activation Mapping (Grad-CAM).

In step S270, the output processing unit 502 executes output such as display. FIG. 14 shows an example of an output display based on such step S270. FIG. 14(A) shows an example of a prediction image marked as a predicted skin loss area in step S250. In the example of FIG. 14(A), the predicted skin loss area is surrounded by a mark such as a frame so that it can be recognized. FIG. 14(B) shows an example of a prediction image that has been processed to visualize the prediction basis in step S260. In the example of FIG. 14(B), it is possible to recognize the basis on which learning model 2 made its prediction by shading (or different colors, etc.).

As described above, according to the skin loss prediction device 5 of the present invention, the predicted skin loss locations are inferred based on the indicated skin loss factor patterns in addition to the indicated skin loss predicted locations, so that it is possible to predict the locations at risk of skin loss with higher accuracy. In addition, because inference is made based on the indicated skin loss factor patterns, it is possible to prevent skin loss prediction factors from becoming a black box.

In addition, the machine learning device 4 according to the present invention can generate a learning model to be used in the skin loss prediction device 5, which can predict areas at risk of skin loss with higher accuracy.

According to the skin loss prediction method of the present invention, the predicted skin loss locations are inferred based on the indicated skin loss factor patterns in addition to the indicated skin loss predicted locations, so that it is possible to predict locations at risk of skin loss with greater accuracy. In addition, because inference is made based on the indicated skin loss factor patterns, it is possible to prevent skin loss prediction factors from becoming a black box.

In addition, the machine learning method according to the present invention can generate a learning model to be used in a skin loss prediction method that can predict areas at risk of skin loss with greater accuracy.

In addition, with the skin-fall prediction device 5 and skin-fall prediction method of the present invention, even inexperienced engineers and workers can gain knowledge based on artificial intelligence similar to the skills of geological experts and recognize areas where skin-fall is likely to occur in advance, allowing the engineers and workers to gain experience while ensuring work safety.

In addition, with the skin fall prediction device 5 and skin fall prediction method according to the present invention, locations where skin fall may occur can be identified simply by taking a photograph with an imaging device, so that even those without advanced skills can observe and monitor the face, thereby reducing the number of skilled technicians required.

In addition, the skin fall prediction device 5 and skin fall prediction method according to the present invention make it possible to check whether the work of removing loose rocks (the work of removing loose rocks, etc.) is sufficient, thereby shortening the work time of removing loose rocks.

In addition, the skin-fall prediction device 5 and skin-fall prediction method according to the present invention thoroughly clean the skin and remove loose stones, improving quality.

In addition, the skin fall prediction device 5 and skin fall prediction method according to the present invention can suggest an appropriate spray thickness, allowing for rational mirror spraying.

In addition, the skin fall prediction device 5 and skin fall prediction method according to the present invention eliminate the need for skin fall prevention equipment, which is economical and improves the efficiency of face work.

Situations similar to those of a tunnel face can also be seen during large-scale cut slope excavation of rock mass, on quarry wall slopes, and on natural slopes. The face fall prediction device 5 and face fall prediction method of the present invention are not limited to mountain tunnel faces, but can also be applied to these face and slopes, and are one of the monitoring methods for preventing face falls and rockfall disasters.

In the embodiment described so far, the learning model 2 is configured by a neural network model (including deep learning) 20 that treats the prediction and judgment of the skin drop of the tunnel face 10 as a classification problem as a machine learning learning method. However, the learning model may be configured by a neural network model that treats it as a regression problem. Furthermore, as long as the correlation between the input data and the output data in the above embodiment is machine-learned from the learning data, other learning methods may be adopted without being limited to the above examples. For example, the learning model may use a recurrent neural network (RNN) or ensemble learning. Furthermore, when it is treated as a regression problem, the learning model may use a statistical learning method, for example, a statistical model such as an autoregressive integrated moving average (ARIMA) model or Bayesian estimation.

Furthermore, each unit of the control unit 40 of the machine learning device 4 or the control unit 50 of the skin fall prediction device 5 in the above embodiment may be realized by having the processor 912 of the computer 900 shown in Figure 11 execute a program.
Cut.

In addition, instead of the aspect of the surface fall prediction device 5 according to the above embodiment, it may be provided in the form of an inference device (inference method or inference program) used to predict and determine the surface fall of the tunnel face 10. In this case, the inference device includes a memory and a processor, and the processor executes a series of processes.

This series of processes includes a data acquisition process for acquiring judgment data including a prediction image 31, and, once the judgment data has been acquired in the data acquisition process, an inference process for inputting the prediction image 31 into a learning model and inferring the predicted location of the skin drop of the tunnel face captured in the prediction image 31.

By providing the inference device (inference method or inference program) as described above, it is possible to easily apply it to various devices compared to implementing the skin fall prediction device 5. It should be obvious to those skilled in the art that when the inference device (inference method or inference program) predicts and determines the skin fall of the tunnel face 10, it may apply the inference method implemented by the inference unit 501 of the skin fall prediction device 5 using the trained learning model 2 generated by the machine learning device 4 according to the above embodiment.

1. Skin loss prediction system,
2, 2a, 2b...Learning model,
3...imaging device,
4...Machine learning device,
5... Skin loss prediction device,
6... Terminal device,
7. Network,
8A: Environmental monitoring device,
8B...Environmental information providing device,
10...Tunnel face,
20...Neural network model,
21...input layer,
22a...first intermediate layer,
22b: second intermediate layer;
23a...first output layer,
23b... second output layer,
23c... third output layer 30... learning image,
31... Prediction image,
40: Control unit,
41...Communication department,
42: learning data storage unit,
43... trained model storage unit,
50: Control unit,
51: Communication unit,
52... trained model storage unit,
80...Learning environment information,
81...Environmental information for judgment,
220a, 220b...convolutional layers,
221a, 221b...pooling layer,
222a, 222b, 222c... fully connected layers,
400, 400a, 400b...learning data acquisition unit,
401, 401a, 401b...machine learning unit,
500, 500a...judgment data acquisition unit,
501, 501a, 501b... Reasoning section,
502: Output processing unit,
900... Computer 910... Bus,
912...Processor,
914...Memory,
916...input device,
917...output device,
918...Display device,
920...Storage device,
922: Communication I/F (interface) unit,
924...external device I/F section,
926: I/O (input/output) device I/F unit,
928...Media input/output unit

Claims (6)

A skin fall prediction device for predicting skin fall at a tunnel face,
A judgment data acquisition unit that acquires judgment data including a prediction image of a tunnel face;
A learned model storage unit that stores a learned model that machine-learns a correlation between input data including a learning image of the tunnel face and output data including data on a predicted skin fall indication point and a skin fall cause pattern indication point included in the learning image;
An inference unit that inputs the judgment data acquired by the judgment data acquisition unit into the learning model and infers a predicted location of the skin fall of the tunnel face captured in the prediction image ,
The prediction image is cut out into a plurality of image data for use, and
Each of the plurality of image data has an overlapping portion with other image data,
Each of the plurality of image data has a weighting coefficient, and the weighting coefficient of the image data is input to the learning model together with the image data;
The overlapping portion includes a plurality of images divided into a grid pattern,
A skin fall prediction device characterized in that the weighting coefficient is higher at the periphery of a tunnel face than at the center . 2. The skin defect prediction device according to claim 1 , wherein the skin defect prediction portion is marked in the prediction image. 3. The skin defect prediction device according to claim 1 , wherein a prediction basis for the predicted skin defect portion is visualized in the prediction image. A machine learning device that generates a learning model used in a skin fall prediction device that predicts skin falls at a tunnel face,
A learning data storage unit that stores a plurality of sets of learning data including input data including learning images of the tunnel face and output data including data on predicted skin fall indication points and skin fall cause pattern indication points included in the learning images;
a machine learning unit that inputs a plurality of sets of the learning data into the learning model to machine-learn the correlation between the input data and the output data;
a trained model storage unit that stores the trained model trained by the machine learning unit ;
The learning image is cut out into a plurality of image data for use, and
Each of the plurality of image data has an overlapping portion with other image data,
Each of the plurality of image data has a weighting coefficient, and the weighting coefficient of the image data is input to the learning model together with the image data;
The overlapping portion includes a plurality of images divided into a grid pattern,
A machine learning device characterized in that the weighting coefficient is higher at the periphery of a tunnel face than at the center . A method for predicting a skin fall at a tunnel face, comprising:
A judgment data acquisition process for acquiring judgment data including a prediction image of a tunnel face;
The judgment data acquired by the judgment data acquisition process is input to a learning model that has been machine-learned to learn the correlation between input data including a learning image of the tunnel face and output data including data related to a skin fall prediction indication point and a skin fall cause pattern indication point included in the learning image, and an inference process is provided to infer the skin fall prediction point of the tunnel face captured in the prediction image ,
The prediction image is cut out into a plurality of image data for use, and
Each of the plurality of image data has an overlapping portion with other image data,
Each of the plurality of image data has a weighting coefficient, and the weighting coefficient of the image data is input to the learning model together with the image data;
The overlapping portion includes a plurality of images divided into a grid pattern,
A method for predicting skin fall, characterized in that the weighting coefficient is higher at the periphery of a tunnel face than at the center . A machine learning method for generating a learning model used in a skin fall prediction method for predicting skin falls at a tunnel face,
a learning data storage step of storing a plurality of sets of learning data in a learning data storage unit, the learning data being composed of input data including learning images of the tunnel face and output data including data on predicted skin-fall indication points and skin-fall factor pattern indication points included in the learning images;
a machine learning process for inputting a plurality of sets of the learning data into the learning model, thereby making the learning model learn the correlation between the input data and the output data by machine learning;
a trained model storage step of storing the trained model trained by the machine learning step in a trained model storage unit ;
The learning image is cut out into a plurality of image data for use, and
Each of the plurality of image data has an overlapping portion with other image data,
Each of the plurality of image data has a weighting coefficient, and the weighting coefficient of the image data is input to the learning model together with the image data;
The overlapping portion includes a plurality of images divided into a grid pattern,
A machine learning method characterized in that the weighting coefficient is higher at the periphery of the tunnel face than at the center .