JP7586486B2 - Structure condition determination device, machine learning device, inference device, structure condition determination method, and machine learning method - Google Patents

Description

The present invention relates to a structure condition determination device, a machine learning device, an inference device, a structure condition determination method, and a machine learning method.

Conventionally, when determining whether or not inspection and maintenance work is required for various structures, or when formulating plans to carry out inspection and maintenance work, the surface of the structure is imaged with an imaging device, and the damage state of the structure is evaluated based on the captured image. For example, Patent Document 1 discloses a device that detects the depth of a crack from an image of the surface of an inspection target, and evaluates the lifespan of the structure according to the depth of the crack.

JP 2009-175110 A

Patent Document 1 discloses that the lifespan of a structure is evaluated according to the depth of cracks that have appeared on the surface of the object being inspected, but the lifespan of a structure is considered to represent the time until the function of the structure is impaired.

Here, when damage occurs to a structure due to deterioration over time, the degree of damage, which indicates the extent of the damage, gradually progresses. Therefore, the manager of the structure must determine whether or not to carry out inspection and maintenance work and develop an implementation plan for the inspection and maintenance work, anticipating not the lifespan of the structure, i.e., the time until the function of the structure is impaired, but the time required for the damage to progress to the next stage. However, the device disclosed in Patent Document 1 cannot calculate the time required for the damage to progress to the next stage, making it difficult to carry out inspection and maintenance work according to the damage level.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide a structure condition determination device, a machine learning device, an inference device, a structure condition determination method, and a machine learning method that facilitate the implementation of inspection and maintenance work according to the degree of damage progression.

In order to achieve the above object, a structure state determination device according to one aspect of the present invention comprises:
A structure condition determination device for determining a damage condition of a structure, comprising:
a determination data acquisition unit that acquires determination data including a determination image in which at least a portion of the structure is captured;
a trained model storage unit that stores a trained model that has been machine-learned to obtain a correlation between input data including a training image in which at least a portion of the structure is captured, and output data including a damage type of a damaged portion included in the training image, a damage progression indicating the progression of the damaged portion when the progression of the damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage; and
The method further includes an inference unit that inputs the judgment data acquired by the judgment data acquisition unit into the learning model and infers the damage type, the damage progression level, and the damage progression time as the damage state of the structure captured in the judgment image.

According to a structure condition determination device according to one aspect of the present invention, determination data including a determination image in which at least a portion of a structure is captured is input to a learning model, and the damage type, damage progression, and damage progression time are inferred as the damage state of the structure captured in the determination image, making it easier to carry out inspection and maintenance work according to the damage progression.

Other issues, configurations and advantages will be made clear in the detailed description of the invention described below.

1 is an overall configuration diagram showing an example of a structure management system 1 according to a first embodiment. FIG. 2 is a block diagram showing an example of a machine learning device 4 according to the first embodiment. FIG. 2 is a data configuration diagram showing an example of learning data according to the first embodiment. 2 is a schematic diagram showing an example of a neural network model 20 applied to a learning model 2 according to the first embodiment. FIG. FIG. 2 is a block diagram showing an example of a structure state determination device 5 according to the first embodiment. 5 is a functional explanatory diagram showing an example of an inference process performed by an inference unit 501 according to the first embodiment. FIG. 9 is a hardware configuration diagram showing an example of a computer 900 constituting each device 3 to 6 of the structure management system 1. FIG. 4 is a flowchart showing an example of a machine learning method performed by the machine learning device 4 according to the first embodiment. 4 is a flowchart showing an example of a structure state determination method performed by the structure state determination device 5 according to the first embodiment. FIG. 11 is a block diagram showing an example of a machine learning device 4 according to a second embodiment. A schematic diagram showing an example of a neural network model 20b applied to a learning model 2a relating to the second embodiment. FIG. 11 is a block diagram showing an example of a structure state determination device 5 according to a second embodiment. FIG. 11 is a functional explanatory diagram showing an example of an inference process performed by an inference unit 501a according to the second embodiment. FIG. 13 is a block diagram showing an example of a machine learning device 4 according to a third embodiment. A schematic diagram showing an example of a neural network model 20b applied to a learning model 2b relating to the third embodiment. FIG. 13 is a block diagram showing an example of a structure state determination device 5 according to a third embodiment. FIG. 13 is a functional explanatory diagram showing an example of an inference process performed by an inference unit 501b according to the third embodiment. 1 is a graph showing the relationship between the damage progression level and damage progression time for each damage type.

The following describes an embodiment of the present invention with reference to the drawings. The following shows a schematic view of the scope of the description necessary to achieve the object of the present invention, and mainly describes the scope of the description necessary to explain the relevant parts of the present invention. The parts that are omitted are based on publicly known technology.

(First embodiment)
1 is an overall configuration diagram showing an example of a structure management system 1 according to a first embodiment. The structure management system 1 is a system for managing the damage state of a structure 10 that is a management target.

The structure management system 1 comprises an imaging device 3 that captures images of at least a portion of the structure 10 to be managed, a machine learning device 4 that operates as the subject of the learning phase of the machine learning, a structure state determination device 5 that operates as the subject of the inference phase of the machine learning, and a terminal device 6 used by the manager and workers of the structure management system 1. The devices 3 to 6 of the structure management system 1 are connected to each other via a network 7 so that they can communicate with each other.

The structure 10 is constructed in various locations by being formed from exterior materials or paving materials such as concrete or fiber-reinforced plastic (FRP). Specific examples of the structure 10 include, but are not limited to, buildings, bridges, solar power generation facilities, wind power generation facilities, dams, railway facilities, roads, chimneys, tunnels, etc.

Various types of damage 100 may occur in each part of the structure 10 due to deterioration over time, and when damage 100 occurs, the degree of damage 100, which indicates the extent of the damage, gradually progresses. Therefore, the structure management system 1 operates as a system for managing the damage state of the structure 10 and providing information regarding the need for inspection and maintenance work, and information regarding the formulation of an implementation plan for inspection and maintenance work.

The imaging device 3 includes a camera configured with an image sensor such as a CMOS sensor or a CCD sensor, and captures images of each part of the structure 10. The imaging device 3 may be a handheld or fixed camera that is operated by an operator to capture images, or may capture images automatically when predetermined imaging conditions are met. The imaging device 3 may also be a camera attached to a drone or robot, etc., that is remotely controlled or autonomously moved to each imaging position of the structure 10 to capture images.

The imaging device 3 captures an image of the structure 10 within the angle of view of the imaging device 3, and outputs the image as digital data based on a predetermined image format. As shown in FIG. 1, the imaging device 3 may be provided separately as an imaging device 3 connected to the machine learning device 4 and an imaging device 3 connected to the structure state determination device 5, or one imaging device 3 may be connected to both the machine learning device 4 and the structure state determination device 5 and used in common. Also, in FIG. 1, for simplicity, one imaging device 3 is connected to each of the machine learning device 4 and the structure state determination device 5, but multiple imaging devices 3 may be connected to each.

The machine learning device 4 is composed of a general-purpose or dedicated computer (see FIG. 7 described later) or the like, and operates as the subject of the learning phase in machine learning. The machine learning device 4 performs machine learning of the learning model 2 using learning data including images (learning images) captured by the imaging device 3. The machine learning device 4 provides the trained learning model 2 to the structure state determination device 5 via the network 7, a recording medium, or the like. Details of the machine learning device 4 will be described later.

The structure state determination device 5 is composed of a general-purpose or dedicated computer (see FIG. 7 described later) or the like, and operates as the subject of the inference phase in machine learning. The structure state determination device 5 uses a learning model 2 that has been trained by the machine learning device 4 to determine the damage state of the structure 10 from an image (determination image) captured by the imaging device 3. Details of the structure state determination device 5 will be described later.

The terminal device 6 is composed of a general-purpose or dedicated computer (see FIG. 7 described later), etc. The terminal device 6 accepts various operational inputs via an input screen in order to prepare learning data, perform machine learning of the learning model 2, determine the damage state of the structure 10, etc. in the structure management system 1, and displays various information via a display screen such as an app or a browser. Also, for the sake of simplification, one terminal device 6 is illustrated in FIG. 1, but a plurality of terminal devices 6 may be provided.

(Machine learning device 4)
2 is a block diagram showing an example of the machine learning device 4 according to the first embodiment. The machine learning device 4 includes a control unit 40, a communication unit 41, a training data storage unit 42, and a trained model storage unit 43.

The control unit 40 functions as a learning data acquisition unit 400 and a machine learning unit 401. The communication unit 41 is connected to external devices (e.g., the imaging device 3, the structure state determination device 5, and the terminal device 6) via the network 7, and functions as a communication interface for transmitting and receiving various types of data.

The learning data acquisition unit 400 is connected to an external device via the communication unit 41 and the network 7, and acquires learning data that is composed of corresponding input data and output data.

The learning data storage unit 42 is a database that stores multiple sets of learning data acquired by the learning data acquisition unit 400. The specific configuration of the database that constitutes the learning data storage unit 42 may be designed as appropriate.

The machine learning unit 401 performs machine learning using the learning data stored in the learning data storage unit 42. That is, the machine learning unit 401 generates the learning model 2 by inputting multiple sets of learning data to the learning model 2 and having the learning model 2 learn the correlation between the input data and output data that constitute the learning data.

The trained model storage unit 43 is a database that stores the trained learning model 2 that has been trained by the machine learning unit 401. The trained model 2 stored in the trained model storage unit 43 is provided to a real system (e.g., a structure state determination device 5) via a network 7 or a recording medium. The trained model 2 may be provided to an external computer (e.g., a server-type computer or a cloud-type computer) and stored in a memory unit of the external computer. Also, although the trained data storage unit 42 and the trained model storage unit 43 are shown as separate memory units in FIG. 2, they may be configured as a single memory unit.

Figure 3 is a data configuration diagram showing an example of learning data according to the first embodiment. The learning data is composed of input data including a learning image 30, and output data including the damage type, damage progression level, and damage progression time of the damaged portion included in the learning image 30.

The learning image 30 included in the input data is an image of at least a portion of the structure 10 captured by the imaging device 3.

In supervised learning, the output data is, for example, called a correct answer label, and is assigned according to the damage state of the structure 10 contained in the learning image 30.

The damage type included in the output data indicates a type of a state when damage 100 occurs in the structure 10, for example, when the state is classified based on the shape or distribution of the damage 100. When the structure 10 is made of concrete, the damage type is classified into a plurality of classes, such as "no damage", "irregular crack", "parallel crack", "lattice crack" and "peeling", including a normal state in which no damage 100 occurs, as shown in FIG. 3. When the structure 10 is made of FRP, the damage type is classified into a plurality of classes, such as "no damage", "wear", and "peeling", including a normal state in which no damage 100 occurs. Note that the damage types are not limited to the definitions and numbers of the above classifications.

The damage progression included in the output data indicates the progression of the damaged portion when the progression of the damage 100 is classified into multiple stages. When damage 100 occurs in the structure 10 from a normal state, the damage progression is classified into three stages, such as mild, moderate, and severe, according to the progression of the damage 100. In this case, the damage progression is classified into multiple classes, such as "mild," "moderate," and "severe." Note that the damage progression is not limited to the definition or number of the above classifications.

The damage progression time included in the output data indicates the time required for the damage 100 progression level to progress to the next stage. For example, if the current damage 100 progression level is mild, the damage 100 progression time indicates the time required for the damage 100 progression level to progress to moderate. In this case, the unit of damage progression time may be days, weeks, months, or years.

In the example of FIG. 3, the output data corresponds to the learning image 30 included in the input data, with the damage type "parallel crack", the damage progression level "medium", and the damage progression time "3 (months)" (in this embodiment, in months). Note that one learning image 30 may include multiple damaged areas.

Figure 4 is a schematic diagram showing an example of a neural network model 20 applied to the learning model 2 according to the first embodiment. The neural network model 20 employs a convolutional neural network (CNN) as a specific method of machine learning. The neural network model 20 includes an input layer 21, an intermediate layer 22, and an output layer 23.

The input layer 21 has neurons whose number corresponds to the number of pixels in the training image 30 as input data, and the pixel value of each pixel is input to each neuron.

The intermediate layer 22 is composed of a convolutional layer 220, a pooling layer 221, and a fully connected layer 222. The convolutional layer 220 and the pooling layer 221 are provided, for example, in multiple alternating layers. The convolutional layer 220 and the pooling layer 221 extract features from the image input via the input layer 21. The fully connected layer 222 converts the two-dimensional array of features extracted from the image by the convolutional layer 220 and the pooling layer 221, for example, by an activation function, and outputs it as a one-dimensional array of feature vectors. Note that the fully connected layer 222 may be provided in multiple layers.

The output layer 23 outputs output data including the results of determining the damaged parts included in the learning image 30 based on the feature vectors output from the fully connected layer 222. In the example of FIG. 4, the output layer 23 has five classes of damage type, three classes of damage progression, and a number of neurons corresponding to the damage progression time.

Between each layer of the neural network model 20, there are synapses that connect the neurons between the layers, and each synapse in the convolutional layer 220 and the fully connected layer 222 of the intermediate layer 22 is assigned a weight.

The machine learning unit 401 inputs the learning data into the neural network model 20, and causes the neural network model 20 to machine-learn the correlation between the input data (learning images 30) and the output data (damage type, damage progression level, and damage progression time). Specifically, the machine learning unit 401 inputs the learning images 30 constituting the learning data as input data to the input layer 21 of the neural network model 20. Note that the machine learning unit 401 may perform predetermined image adjustments (e.g., image format, image size, image filter, image mask, etc.) on the learning images 30 as preprocessing when inputting the learning images 30 to the input layer 21.

The machine learning unit 401 repeatedly adjusts the weights associated with each synapse (backpropagation) so that the evaluation value of the error function becomes smaller, using an error function that compares the output data (determination results of the damage type, damage progression level, and damage progression time) output from the output layer 23 as an inference result with the output data (correct labels of the damage type, damage progression level, and damage progression time) that constitutes the learning data. Then, when the machine learning unit 401 determines that a predetermined learning termination condition, such as the above series of processes being repeated a predetermined number of times or the evaluation value of the error function being smaller than an allowable value, is satisfied, the machine learning is terminated, and the neural network model 20 at that time (a group of weight parameters consisting of all the weights associated with each synapse) is stored in the trained model storage unit 43 as the trained learning model 2. When performing machine learning of the learning model 2, the machine learning unit 401 may adopt, for example, online learning, batch learning, mini-batch learning, etc. as a method for adjusting weights, and may adopt, for example, a hold-out method, cross-validation, etc. as a method for dividing multiple sets of learning data into training data and test data and evaluating the learning model 2, and may determine that the misclassification rate is minimized as a predetermined learning termination condition.

(Structure state determination device 5)
5 is a block diagram showing an example of a structure state determination device 5 according to the first embodiment. The structure state determination device 5 includes a control unit 50, a communication unit 51, and a trained model storage unit 52.

The control unit 50 functions as a judgment data acquisition unit 500, an inference unit 501, and an output processing unit 502. The communication unit 51 is connected to an external device (e.g., the imaging device 3, the machine learning device 4, and the terminal device 6, etc.) via the network 7, and functions as a communication interface for transmitting and receiving various types of data.

The judgment data acquisition unit 500 is connected to an external device via the communication unit 51 and the network 7, and acquires judgment data including a judgment image 31 in which at least a portion of the structure 10 is captured.

The inference unit 501 inputs the judgment data acquired by the judgment data acquisition unit 500 into the learning model 2, and performs an inference process (see FIG. 6 described later) to infer the damage type, damage progression, and damage progression time of the damaged portion included in the judgment image 31 as the damage state of the structure 10.

The trained model storage unit 52 is a database that stores trained learning models 2 used in the inference process of the inference unit 501. The number of trained models 2 stored in the trained model storage unit 52 is not limited to one, and multiple trained models with different conditions, such as machine learning methods and types of exterior materials or coating materials that form the structure 10, may be stored and selectively available. The trained model storage unit 52 may be substituted by a storage unit of an external computer (e.g., a server-type computer or a cloud-type computer), and in that case, the inference unit 501 may perform the above-mentioned inference process by accessing the external computer.

The output processing unit 502 performs output processing for outputting the judgment result of the damage state of the structure 10 inferred by the inference unit 501. As a specific output means for outputting the judgment result, various means can be adopted. For example, the output processing unit 502 may transmit the judgment result to the terminal device 6, further display it on a screen, or may store the judgment result in a storage unit of the structure state judgment device 5.

Figure 6 is a functional explanatory diagram showing an example of inference processing by the inference unit 501 according to the first embodiment.

The judgment image 31 included in the judgment data is an image of at least a portion of the structure 10 captured by the imaging device 3. The judgment image 31 corresponds to the learning image 30 when the machine learning device 4 trains the learning model 2 by machine learning.

The learning model 2 here is a learning model 2 in which the weighting parameter group has been adjusted (trained) by using multiple sets of learning data to machine-learn the correlation between the learning image 30 and the damage type, damage progression, and damage progression time of the damaged part included in the learning image 30 by the machine learning device 4. Therefore, the inference unit 501 inputs the judgment image 31 included in the judgment data to the learning model 2, and infers the damage type, damage progression, and damage progression time of the damaged part included in the judgment image 31.

At this time, the damage type is output as a score (confidence) for each class, and when the damage type classification is expressed as five classes (no damage, irregular crack, parallel crack, lattice crack, and peeling), the scores for each class are output, for example, as "0.02", "0.10", "0.15", "0.95", and "0.12". Any method can be used for using the scores, and as shown in FIG. 6, the class with the highest score (in the above example, the lattice crack with a score of "0.95") may be used as the judgment result, or if the score of a specific class exceeds a specific score reference value (in the above example, the score of the lattice crack class, "0.95", exceeds the score reference value of "0.80"), that class may be used as the judgment result.

The damage progression is output as a score (confidence) for each class, and when the damage progression is classified into three classes (mild, moderate, and severe), the scores for each class are output, for example, as "0.92", "0.14", and "0.08". Any method may be used to use the scores, and as shown in FIG. 6, the class with the highest score (in the above example, mild with a score of "0.92") may be used as the judgment result, or if the score of a specific class exceeds a specific score reference value (in the above example, the score of the mild class, "0.92", exceeds the score reference value of "0.75"), that class may be used as the judgment result.

The damage progression time is output as a numerical value in a specified unit (days, weeks, months, or years). If the damage type is judged to be "lattice crack" and the damage progression is judged to be "moderate," the time required for the lattice crack to progress from "moderate" to "severe" is output as "6 (months)" (in this embodiment, this is in months).

The results of the damage state judgment for the judgment image 31 are preferably stored in the trained model storage unit 52 or another storage device (not shown), and past judgment results may be used as learning data for online learning or re-learning, for example, to further improve the inference accuracy of the trained learning model 2.

(Configuration of computer 900)
FIG. 7 is a hardware configuration diagram showing an example of a computer 900 constituting each of the devices 3 to 6 of the structure management system 1. As shown in FIG.

Each of the devices 3 to 6 in the structure management system 1 is composed of a general-purpose or dedicated computer 900. The computer 900 has as its main components a bus 910, a processor 912, a memory 914, an input device 916, an output device 917, a display device 918, a storage device 920, a communication I/F (interface) unit 922, an external device I/F unit 924, an I/O (input/output) device I/F unit 926, and a media input/output unit 928. Note that the above components may be omitted as appropriate depending on the application for which the computer 900 is used.

The processor 912 is composed of one or more arithmetic processing devices (such as a CPU (Central Processing Unit), MPU (Micro-processing unit), DSP (Digital signal processor), GPU (Graphics Processing Unit)) and operates as a control unit that controls the entire computer 900. The memory 914 stores various data and programs 930, and is composed of, for example, a volatile memory (DRAM, SRAM, etc.) that functions as a main memory, and a non-volatile memory (ROM, flash memory, etc.).

The input device 916 is, for example, a keyboard, a mouse, a numeric keypad, an electronic pen, etc., and functions as an input unit. The output device 917 is, for example, a sound (audio) output device, a vibration device, etc., and functions as an output unit. The display device 918 is, for example, a liquid crystal display, an organic EL display, electronic paper, a projector, etc., and functions as an output unit. The input device 916 and the display device 918 may be integrated, such as a touch panel display. The storage device 920 is, for example, a hard disk drive (HDD), a solid state drive (SSD), etc., and functions as a memory unit. The storage device 920 stores various data necessary for the execution of the operating system and the program 930.

The communication I/F unit 922 is connected to a network 940 (which may be the same as the network 7 in FIG. 1) such as the Internet or an intranet by wire or wirelessly, and functions as a communication unit that transmits and receives data to and from other computers according to a predetermined communication standard. The external device I/F unit 924 is connected to an external device 950 such as a camera, printer, scanner, or reader/writer by wire or wirelessly, and functions as a communication unit that transmits and receives data to and from the external device 950 according to a predetermined communication standard. The I/O device I/F unit 926 is connected to an I/O device 960 such as various sensors and actuators, and functions as a communication unit that transmits and receives various signals and data, such as detection signals from sensors and control signals to actuators, between the I/O device 960. The media input/output unit 928 is composed of a drive device such as a DVD (Digital Versatile Disc) drive or a CD (Compact Disc) drive, and reads and writes data to media (non-transient storage media) 970 such as DVDs and CDs.

In the computer 900 having the above configuration, the processor 912 calls up the program 930 stored in the storage device 920 into the memory 914, executes it, and controls each unit of the computer 900 via the bus 910. The program 930 may be stored in the memory 914 instead of the storage device 920. The program 930 may be recorded in the medium 970 in an installable file format or an executable file format, and provided to the computer 900 via the media input/output unit 928. The program 930 may be provided to the computer 900 by downloading it via the network 940 via the communication I/F unit 922. In addition, the computer 900 may realize the various functions realized by the processor 912 executing the program 930 using hardware such as a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

The computer 900 is, for example, a stationary computer or a portable computer, and is an electronic device of any form. Depending on the use of the computer 900, the computer 900 may be a client-type computer or an edge-type computer, or may be a server-type computer or a cloud-type computer.

(Machine learning methods)
FIG. 8 is a flowchart showing an example of a machine learning method performed by the machine learning device 4 according to the first embodiment.

First, in step S100, the learning data acquisition unit 400 prepares a desired number of pieces of learning data as a preliminary step for starting machine learning, and stores the prepared learning data in the learning data storage unit 42. The number of pieces of learning data to be prepared here may be set in consideration of the inference accuracy required for the learning model 2 that is ultimately obtained.

Several methods can be adopted for preparing the learning data. For example, the worker uses the imaging device 3 to capture the learning image 30 at any imaging location of the structure 10, and uses the terminal device 6 to record the imaging date and time and the imaging location, and if damage 100 occurs, the damage type and damage progression of the damage 100 are recorded. Then, when the worker captures a new learning image 30 at the same imaging location, if the worker determines that the damage progression of the damage 100 has progressed to the next stage, the learning data is prepared by associating the damage progression time (for example, the elapsed time from the previous imaging date and time to the current imaging date and time) with the previously captured learning image 30. Then, by repeating such operations, it is possible to prepare multiple sets of learning data. In addition, as another method, it is also possible to obtain learning data by, for example, performing an acceleration test in which damage 100 is intentionally generated in a test specimen of the structure 10. Furthermore, multiple sets of learning data are prepared, consisting of input data and output data when no damage 100 has occurred, i.e., when the structure 10 is in a normal state (for example, the damage type of the output data in this case is "no damage"), as well as when damage 100 has occurred.

Next, in step S110, the machine learning unit 401 prepares a pre-learning learning model 2 in order to start machine learning. The pre-learning learning model 2 prepared here employs the convolutional neural network illustrated in FIG. 4, and the weights of each synapse are set to an initial value. Each neuron in the input layer 21 is associated with each pixel of the learning image 30 as input data constituting the learning data. Each neuron in the output layer 23 is associated with a damage type, damage progression level, and damage progression time as output data constituting the learning data.

Next, in step S120, the machine learning unit 401 acquires, for example, one set of learning data randomly from the multiple sets of learning data stored in the learning data storage unit 42.

Next, in step S130, the machine learning unit 401 inputs the input data included in the set of learning data to the input layer of the prepared learning model 2 before learning (or during learning). As a result, output data is output as an inference result from the output layer of the learning model 2, but this output data was generated by the learning model 2 before learning (or during learning). Therefore, in the pre-learning (or during learning) state, the output data output as an inference result indicates information different from the output data (correct answer label) included in the learning data.

Next, in step S140, the machine learning unit 401 performs machine learning by comparing the output data (correct label) included in the set of learning data acquired in step S120 with the output data output from the output layer as an inference result in step S130 and adjusting the weight of each synapse. In this way, the machine learning unit 401 causes the learning model 2 to learn the correlation between the input data and the output data.

Next, in step S150, the machine learning unit 401 determines whether a predetermined learning termination condition has been met, for example, based on the evaluation value of an error function based on the inference result and the output data (correct answer label) included in the learning data, or the remaining amount of unlearned learning data stored in the learning data storage unit 42.

If, in step S150, the machine learning unit 401 determines that the learning termination condition is not satisfied and machine learning is to be continued (No in step S150), the process returns to step S120, and steps S120 to S140 are performed multiple times on the learning model 2 being trained using untrained learning data. On the other hand, in step S150, if the machine learning unit 401 determines that the learning termination condition is satisfied and machine learning is to be terminated (Yes in step S150), the process proceeds to step S160.

In step S160, the machine learning unit 401 stores the trained learning model 2 (adjusted weight parameter group) trained by adjusting the weights associated with each synapse in the trained model storage unit 43, and ends the series of machine learning methods shown in FIG. 8. In the machine learning method, step S100 corresponds to the training data storage process, steps S110 to S150 correspond to the machine learning process, and step S160 corresponds to the trained model storage process. Note that in the series of machine learning methods shown in FIG. 8, the case where online learning is adopted as a method for adjusting the weights has been described, but batch learning or mini-batch learning (for example, 100 sets of training data units) may also be adopted. In addition, when multiple sets of training data are divided into training data and test data, the series of machine learning methods shown in FIG. 8 may be executed using the training data. Furthermore, it may be determined whether a predetermined learning end condition is met based on the misjudgment rate.

As described above, the machine learning device 4 and machine learning method according to this embodiment can provide a learning model 2 that can infer (determine) the damage state of the structure 10 with high accuracy from a learning image 30 in which at least a portion of the structure 10 is captured by the imaging device 3.

(Structure condition determination method)
Fig. 9 is a flowchart showing an example of a structure state determination method by the structure state determination device 5 according to the first embodiment. The series of structure state determination methods shown in Fig. 9 are repeatedly executed at a predetermined timing by the structure state determination device 5. The predetermined timing may be any timing, and may be, for example, when the imaging device 3 captures a new determination image 31, or when a predetermined event occurs (when a determination operation by a manager or worker is accepted from the terminal device 6, when a determination command is received from the structure management system 1, etc.).

First, in step S200, the structure 10 is imaged by the imaging device 3, and the judgment image 31 is transmitted to the structure state judgment device 5, whereby the judgment data acquisition unit 500 acquires judgment data including the judgment image 31.

Next, in step S210, the inference unit 501 performs a predetermined preprocessing on the judgment data and inputs it to the input layer of the trained learning model 2, and obtains the scores for each class of damage type, the scores for each class of damage progression, and the numerical values of damage progression time output as inference results from the output layer of the learning model 2.

Next, in step S220, as an example of post-processing of supervised learning, the inference unit 501 determines the damage type, damage progression, and damage progression time of the damaged part included in the determination image 31 by, for example, determining the highest class among the scores for each damage type class as the damage type and the highest class among the scores for each damage progression class as the damage progression.

Next, in step S230, the output processing unit 502 determines whether the result of the damage type determination by the inference unit 501 is "no damage" or not. If it determines that it is "no damage," the process proceeds to step S240, and if it does not determine that it is "no damage," the process proceeds to step S250.

Then, in step S240, the output processing unit 502 outputs information indicating "no damage." Note that step S240 may be omitted.

On the other hand, in step S250, the output processing unit 502 outputs information indicating the damage type, the damage progression level, and the damage progression time as information indicating "damage present," and ends the series of steps in the structure state determination method shown in Figure 9. In the structure state determination method, step S200 corresponds to the determination data acquisition process, steps S210 to S220 correspond to the inference process, and steps S230 to S250 correspond to the output processing process.

As described above, according to the structure condition determination device 5 and structure condition determination method of this embodiment, determination data including a determination image 31 in which at least a portion of a structure 10 is captured is input to the learning model 2, and the damage type, damage progression, and damage progression time are inferred as the damage state of the structure 10 captured in the determination image 31, making it easier to carry out inspection and maintenance work according to the damage progression.

Second Embodiment
Fig. 10 is a block diagram showing an example of a machine learning device 4 according to the second embodiment. Fig. 11 is a schematic diagram showing an example of a neural network model 20a applied to a learning model 2a according to the second embodiment. Fig. 12 is a block diagram showing an example of a structure state determination device 5 according to the second embodiment. Fig. 13 is a functional explanatory diagram showing an example of an inference process by an inference unit 501a according to the second embodiment.

The learning model 2a used in the machine learning device 4 and the structure state determination device 5 according to the second embodiment is obtained by changing the contents of the input data compared to the learning model 2 according to the first embodiment. Specifically, in the learning model 2a according to the second embodiment, the input data constituting the learning data further includes learning environment information 80 indicating the environment of the imaging location where the learning image 30 was captured, and accordingly, the determination data further includes determination environment information 81 indicating the environment of the imaging location where the determination image 31 was captured. Since the basic configuration and operation of the machine learning device 4 and the structure state determination device 5 according to the second embodiment are the same as those of the first embodiment, the following description will focus on the differences from the first embodiment.

The learning environment information 80 and the judgment environment information 81 are information on at least one of the amount of ultraviolet rays, temperature, humidity, wind speed, ocean waves, lightning, climate, acidity value of acid rain, amount of substances contained in acid rain, amount of airborne salt, and amount of radiation. The learning environment information 80 and the judgment environment information 81 are preferably calculated, for example, as average values or cumulative values over a specified observation period.

The learning environment information 80 and the judgment environment information 81 are observation data observed by various environmental sensors equipped in the environmental observation device 8A, and the various environmental sensors may be installed in the structure 10, or may be installed in each predetermined region and observed as representative values representing each region. Furthermore, the learning environment information 80 and the judgment environment information 81 are provided data provided by an environmental information providing device 8B operated by the Japan Meteorological Agency, an environmental information provider, or the like, and may be provided data corresponding to the position or region of the structure 10.

The learning data acquisition unit 400a of the machine learning device 4 acquires learning data consisting of input data including the learning image 30 and the learning environment information 80, and output data including the damage type, damage progression level, and damage progression time of the damaged portion included in the learning image 30.

The machine learning unit 401a inputs multiple sets of learning data into the learning model 2a, and causes the learning model 2a to learn by machine learning the correlation between the input data (learning images 30 and learning environment information 80) that constitute the learning data and the output data (damage type, damage progression, and damage progression time).

In the neural network model 20a (see FIG. 11) applied to the learning model 2a, the learning image 30 is input to the input layer 21, and the learning environment information 80 is input to the fully connected layer 222. Therefore, the fully connected layer 222 further has at least one neuron to which the learning environment information 80 (determination environment information 81) is input. The learning environment information 80 (determination environment information 81) may be a part or all of the information exemplified above, or may further include other information, and the number of neurons in the fully connected layer 222 is set according to the number of pieces of learning environment information 80 (determination environment information 81). In addition, when the learning environment information 80 (determination environment information 81) is input to the fully connected layer 222, the value may be input as it is, or a normalized value may be input, or, for example, a categorical variable or one-hot encoding may be introduced and the converted value may be input.

The judgment data acquisition unit 500a of the structure state judgment device 5 acquires judgment data including the judgment image 31 and judgment environment information 81.

The inference unit 501a inputs the judgment data (judgment image 31 and judgment environment information 81) acquired by the judgment data acquisition unit 500a into the learning model 2a, and infers the damage type, damage progression degree, and damage progression time of the damaged part contained in the judgment image 31 as the damage state of the structure 10. At that time, the judgment image 31 is input to the input layer 21, and the judgment environment information 81 is input to the fully connected layer 222.

As described above, according to this embodiment, by adding the learning environment information 80 and the judgment environment information 81 to the learning image 30 and the judgment image 31, respectively, and carrying out machine learning of the learning model 2a, the influence of the learning environment information 80 and the judgment environment information 81 on the damage state of the structure 10 is taken into account, thereby improving the judgment accuracy of the learning model 2a.

Third Embodiment
Fig. 14 is a block diagram showing an example of a machine learning device 4 according to the third embodiment. Fig. 15 is a schematic diagram showing an example of a neural network model 20b applied to a learning model 2b according to the third embodiment. Fig. 16 is a block diagram showing an example of a structure state determination device 5 according to the third embodiment. Fig. 17 is a functional explanatory diagram showing an example of an inference process by an inference unit 501b according to the third embodiment.

A learning model 2b used in the machine learning device 4 and structure state determination device 5 according to the third embodiment is obtained by changing the contents of output data compared to the learning model 2 according to the first embodiment. Specifically, in the learning model 2b according to the third embodiment, the output data constituting the learning data includes a plurality of damage progression times indicating the times required for the damage progression degree to progress to each of the following plurality of stages. Since the basic configuration and operation of the machine learning device 4 and structure state determination device 5 according to the third embodiment are similar to those of the first embodiment, the following description will focus on the differences from the first embodiment.

The learning data acquisition unit 400b of the machine learning device 4 acquires learning data consisting of input data including the learning image 30 and output data including the damage type, damage progression level, and multiple damage progression times of the damaged portion included in the learning image 30.

The machine learning unit 401b inputs multiple sets of learning data into the learning model 2b, and allows the learning model 2b to learn by machine learning the correlation between the input data (learning images 30) that constitute the learning data and the output data (damage type, damage progression level, and multiple damage progression times).

In this case, in the neural network model 20b (see FIG. 15) applied to the learning model 2b, the multiple damage progression times include, for example, the damage progression time required for the damage progression level to progress from "start" to "mild", the damage progression time required for the damage progression level to progress from "mild" to "moderate", and the damage progression time required for the damage progression level to progress from "moderate" to "severe", and are output from each neuron in the output layer 23. Therefore, the output layer 23 has five classes of damage types, three classes of damage progression levels, and a number of neurons corresponding to the three damage progression times.

Fig. 18 is a graph showing the relationship between the damage progression level and the damage progression time for each damage type. In Fig. 18, for four classes of damage types, namely "irregular crack", "parallel crack", "lattice crack" and "peeling", the horizontal axis represents time and the vertical axis represents the damage progression level, and three damage progression times as the damage progression level progresses are illustrated as a time-dependent deterioration graph.

As described above, the learning data is composed of input data including the learning image 30, and output data including the damage type, damage progression degree, and multiple damage progression times of the damaged parts included in the learning image 30, so the machine learning unit 401b trains the learning model 2b to machine learn the time-dependent deterioration graphs for each of the above damage types.

The inference unit 501b of the structure state determination device 5 inputs the determination data (determination image 31) acquired by the determination data acquisition unit 500 into the learning model 2b, and infers the damage type, damage progression level, and multiple damage progression times of the damaged parts included in the determination image 31 as the damage state of the structure 10.

As described above, according to this embodiment, by performing machine learning of the learning model 2b using learning data including multiple damage progression times as output data, the damage type, damage progression level, and multiple damage progression times are inferred as the damage state of the structure 10 captured in the assessment image 31, so that the progress of the damage progression can be determined in more detail.

Other Embodiments
The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention. All such modifications are included in the technical concept of the present invention.

For example, the second embodiment and the third embodiment may be combined, and the output data constituting the learning data according to the second embodiment may include a plurality of damage progression times as in the third embodiment. In this case, the machine learning unit of the machine learning device 4 machine-learns the correlation between the input data (the learning image 30 and the learning environment information 80) and the output data (the damage type, the damage progression level, and the plurality of damage progression times) into a learning model. In addition, the inference unit of the structure state determination device 5 inputs the determination data (the determination image 31 and the determination environment information 81) into the trained learning model, thereby inferring the damage type, the damage progression level, and the plurality of damage progression times of the damaged portion included in the determination image 31 as the damage state of the structure 10.

In the above embodiment, the learning models 2, 2a, and 2b are constructed using neural network models (including deep learning) 20, 20a, and 20b that treat the determination of the damage state of the structure 10 as a classification problem as a machine learning learning method. However, the learning models may be constructed using neural network models that treat it as a regression problem. Furthermore, as long as the correlation between the input data and the output data in the above embodiment is machine-learned from the learning data, other learning methods may be adopted without being limited to the above examples. For example, the learning model may use a recurrent neural network (RNN) or ensemble learning. In addition, when treating it as a regression problem, the learning model may use a statistical learning method, such as an autoregressive integrated moving average (ARIMA) model or a statistical model such as Bayesian estimation.

Furthermore, each part of the control unit 40 of the machine learning device 4 or the control unit 50 of the structure state determination device 5 in the above embodiment may be realized by having the processor 912 of the computer 900 shown in Figure 7 execute a program.
Cut.

In addition, instead of the structure state determination device 5 according to the above embodiment, the inference device may be provided in the form of an inference device (inference method or inference program) used to determine the damage state of the structure 10. In this case, the inference device includes a memory and a processor, and the processor executes a series of processes. The series of processes includes a data acquisition process for acquiring determination data including the determination image 31, and an inference process for inferring the damage type, damage progression, and damage progression time of the damaged part included in the determination image 31 as the damage state of the structure 10 captured in the determination image 31 when the determination data is acquired in the data acquisition process. By providing the inference device (inference method or inference program) in the form of the above, it is possible to easily apply it to various devices compared to the case where the structure state determination device 5 is implemented. It should be obvious to those skilled in the art that when the inference device (inference method or inference program) determines the damage state of the structure 10, the inference method implemented by the inference units 501, 501a, and 501b of the structure state determination device 5 may be applied using the trained learning models 2, 2a, and 2b generated by the machine learning device 4 according to the above embodiment.

1... structure management system, 2, 2a, 2b... learning model, 3... imaging device,
4...machine learning device, 5...structure state determination device, 6...terminal device, 7...network,
8A...environmental observation device, 8B...environmental information providing device, 10...structure,
20, 20a, 20b... neural network model,
21...input layer, 22...intermediate layer, 23...output layer,
30 ... learning image, 31 ... judgment image,
40: control unit, 41: communication unit, 42: learning data storage unit, 43: learned model storage unit, 50: control unit, 51: communication unit, 52: learned model storage unit,
80 ... learning environment information, 81 ... judgment environment information, 100 ... damage,
220...convolutional layer, 221...pooling layer, 222...fully connected layer,
400, 400a, 400b...learning data acquisition unit,
401, 401a, 401b...machine learning unit,
500, 500a...judgment data acquisition unit, 501, 501a, 501b...inference unit,
502: Output processing unit, 900: Computer

Claims (11)

A structure condition determination device for determining a damage condition of a structure, comprising:
a determination data acquisition unit that acquires determination data including a determination image in which at least a portion of the structure is captured;
a trained model storage unit that stores a trained model that has been machine-learned to obtain a correlation between input data including a training image in which at least a portion of the structure is captured, and output data including a damage type of a damaged portion included in the training image, a damage progression indicating the progression of the damaged portion when the progression of the damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage; and
an inference unit that inputs the judgment data acquired by the judgment data acquisition unit into the learning model and infers the damage type, the damage progression level, and the damage progression time as the damage state of the structure captured in the judgment image,
Structural condition determination device. The output data is
the damage type, the damage progression level, and a plurality of damage progression times indicating times required for the damage progression level to progress to a plurality of subsequent stages,
The inference unit is
Inferring the damage type, the damage progression degree, and a plurality of damage progression times as the damage state;
The structure state determination device according to claim 1 . The determination data is
Further, the determination-use environment information indicates an environment of an imaging location where the determination-use image is captured.
The input data is
Further, learning environment information indicating an environment of an imaging location where the learning image was captured is included.
The structure state determination device according to claim 1 or 2. The judgment environment information and the learning environment information are
The information is information on at least one of the amount of ultraviolet rays, temperature, humidity, wind speed, ocean waves, lightning, climate, the acidity value of acid rain, the amount of substances contained in the acid rain, the amount of airborne salt, and the amount of radiation.
The structure state determination device according to claim 3 . A machine learning device that generates a learning model used in a structure condition determination device that determines a damage state of a structure,
a learning data storage unit that stores a plurality of sets of learning data, each set including input data including a learning image in which at least a portion of the structure is captured, and output data including a damage type of a damaged portion included in the learning image, a damage progression indicating the progression of the damaged portion when the progression of the damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage;
a machine learning unit that inputs a plurality of sets of the learning data into the learning model to machine-learn the correlation between the input data and the output data;
and a trained model storage unit that stores the trained model trained by the machine learning unit.
Machine learning device. The output data is
The damage type, the damage progression level, and a plurality of damage progression times indicating each time required for the damage progression level to progress to each of the next or subsequent stages,
The structure state determination device according to claim 5 . The input data is
Further, learning environment information indicating an environment of an imaging location where the learning image was captured is included.
The machine learning device according to claim 5 or 6. The learning environment information is
The information is information on at least one of the amount of ultraviolet rays, temperature, humidity, wind speed, ocean waves, lightning, climate, the acidity value of acid rain, the amount of substances contained in the acid rain, the amount of airborne salt, and the amount of radiation.
The machine learning device according to claim 7. An inference device used to determine a damage state of a structure, comprising:
The inference device comprises:
A memory and a processor,
The processor,
A data acquisition process for acquiring determination data including a determination image in which at least a portion of the structure is captured;
When the determination data is acquired in the data acquisition process, an inference process is executed to infer, as the damage state of the structure captured in the determination image, a damage type of the damaged portion included in the determination image, a damage progression indicating the progression of the damaged portion when the progression of the damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage.
Inference device. A method for determining a damage state of a structure, comprising:
a determination data acquisition step of acquiring determination data including a determination image in which at least a portion of the structure is captured;
an inference process for inputting the judgment data acquired by the judgment data acquisition process into a learning model that has been machine-learned to learn a correlation between input data including a learning image in which at least a part of the structure is captured, and output data including a damage type of a damaged portion included in the learning image, a damage progression indicating the progression of the damaged portion when the progression of damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage, and inferring the damage type, the damage progression, and the damage progression time as the damage state of the structure captured in the judgment image.
Method for determining the condition of a structure. A machine learning method for generating a learning model used in a structure condition determination method for determining a damage state of a structure, comprising:
a learning data storage process for storing in a learning data storage unit a plurality of sets of learning data, the learning data being composed of input data including learning images in which at least a portion of the structure is captured, and output data including a damage type of a damaged portion included in the learning image, a damage progression indicating the progression of the damaged portion when the progression of damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage;
a machine learning process for inputting a plurality of sets of the learning data into the learning model, thereby making the learning model learn the correlation between the input data and the output data by machine learning;
and a learned model storage step of storing the learned model machine-learned by the machine learning step in a learned model storage unit.
Machine learning methods.

Images