JP7559247B2 - Sewer pipe abnormality diagnosis support system, client machine, server machine for sewer pipe abnormality diagnosis support system, and related method - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applied July 27, 2020 Disclosure in the publication "57th Sewerage Research Presentation Conference Lecture Collection"

The present invention relates to a system for assisting in the diagnosis of abnormalities (malformations) in sewer pipes, a client machine, a server machine, and related methods that can be used in the system.

As the number of sewer pipes (culverts) exceeding their standard service life increases, and the risk of road collapses and other problems rises, planned maintenance of sewer pipes is more important than ever. Therefore, in order to understand the condition of aging pipes, which will rapidly increase in the future, it is essential to not only increase the daily progress of pipe inspections, but also to improve the speed at which abnormalities (faults) are identified within the pipes. Diagnosis using images is thought to be effective in quickly identifying abnormalities.

Patent Document 1 describes an image diagnostic system that uses a cross-sectional area determined from the deformation position as one diagnostic index and image statistics of a gray-scale gradient in a direction perpendicular to a priori determined crack direction as the other diagnostic index in an image diagnosis of a sewer piping network, diagnosing abnormalities in the sewer piping network based on the axial distribution of the diagnostic indexes that are associated in advance and actual abnormal events. However, the system in Patent Document 1 does not assume that the system will accept feedback such as changes from the user regarding the image diagnosis results obtained by the system, and does not provide a configuration for improving the diagnostic accuracy by continuing to operate the system, nor does it provide a configuration for analyzing abnormal locations from images inside the sewer pipe taking into account the pipe type, pipe condition, etc.

JP 2017-194813 A

In view of the above, the present invention aims to provide a system that assists in diagnosing abnormalities in sewer pipes by using video or still images taken inside the pipes, which can first display the system's judgment results and then accept registration of changes from users, etc., as well as a client machine, a server machine, and related methods that can be used for the system.

In order to solve the above problems, the present invention provides a sewer pipe abnormality diagnosis support system that supports the diagnosis of abnormalities in a sewer pipe by using photographed images, which are videos or still images, taken inside the sewer pipe, and includes a photographed image capture unit that captures the photographed images inside the sewer pipe, an unfolded image generation unit that generates an unfolded image of the inside of the sewer pipe from the photographed images, an abnormality location determination unit that determines abnormal locations inside the sewer pipe from the unfolded image, a determination result display unit that displays locations determined to be abnormal by the abnormality location determination unit as determination results, and a change registration acceptance unit that accepts the registration of changes to the determination result.

The abnormality determination unit may include an image analysis unit, which may include a pipe type and investigation condition receiving unit that receives input of the pipe type and investigation conditions, a binarization parameter setting unit that sets binarization parameters according to the pipe type and investigation conditions, an image conversion unit that converts the unfolded image into a binary image using the binarization parameters, and a binary image analysis unit that analyzes the binary image to determine abnormalities.

The abnormality location determination unit may include a diagnostic model determination unit, which may include a teacher data storage unit that stores an unfolded image of the inside of the sewer pipe and abnormal locations in the unfolded image, a diagnostic model generation unit that uses the unfolded image and the abnormal locations in the unfolded image stored in the teacher data storage unit as teacher data and generates a diagnostic model by machine learning, in which the unfolded image is used as input and the abnormal locations in the unfolded image are used as output, and an abnormality location model determination unit that uses the diagnostic model generated by the diagnostic model generation unit to determine abnormal locations inside the sewer pipe from the unfolded image generated by the unfolded image generation unit.

The diagnostic model determination unit may further include an unfolded image classification unit that classifies the unfolded images by clustering prior to the abnormality area model determination unit determining the abnormality area.

The teacher data storage unit may further store the pipe type corresponding to the unfolded image of the inside of the sewer pipe, and the above system may further include a pipe type estimation model generation unit that uses the unfolded image stored in the teacher data storage unit and the pipe type corresponding to the unfolded image as teacher data to generate a pipe type estimation model by machine learning, with the unfolded image as input and the pipe type corresponding to the unfolded image as output, and a pipe type estimation unit that uses the pipe type estimation model generated by the pipe type estimation model generation unit to estimate the pipe type corresponding to the unfolded image generated by the unfolded image generation unit.

The diagnostic model generation unit may update the diagnostic model using machine learning based on the changes to the judgment results accepted for registration by the change registration acceptance unit.

The captured image acquisition unit, the unfolded image generation unit, the judgment result display unit, and the change registration acceptance unit may be embodied by a client machine, and of the diagnostic model judgment units included in the abnormality location judgment unit, the teacher data storage unit and the diagnostic model generation unit may be embodied by a server machine, and the abnormality location model judgment unit may be embodied by the client machine.

The above system may further include a report output unit that outputs the results of a sewer pipe abnormality diagnosis as a report based on the judgment results by the abnormality location judgment unit and the changes to the judgment results accepted for registration by the change registration acceptance unit.

The present invention also provides a client machine for a sewer pipe abnormality diagnosis support system that supports the diagnosis of abnormalities in sewer pipes using photographed images, which are video or still images, taken inside the sewer pipe, the client machine including: a photographed image capture unit that captures the photographed image inside the sewer pipe; an unfolded image generation unit that generates an unfolded image of the inside of the sewer pipe from the photographed image; a client-side abnormality location determination unit that determines abnormal locations inside the sewer pipe from the unfolded image; a determination result display unit that displays locations determined to be abnormal by the client-side abnormality location determination unit as determination results; and a change registration acceptance unit that accepts the registration of changes to the determination result.

The client-side abnormality determination unit may include an image analysis unit, which may include a pipe type and investigation condition receiving unit that receives input of the pipe type and investigation conditions, a binarization parameter setting unit that sets binarization parameters according to the pipe type and investigation conditions, an image conversion unit that converts the unfolded image into a binary image using the binarization parameters, and a binary image analysis unit that analyzes the binary image to determine abnormalities.

The client-side abnormality location determination unit may include a client-side diagnostic model determination unit, which may include a diagnostic model receiving unit that receives a diagnostic model that takes the unfolded image as input and the abnormal location in the unfolded image as output, the diagnostic model being generated by machine learning using the unfolded image and the abnormal location in the unfolded image as training data, and an abnormality location model determination unit that uses the diagnostic model received by the diagnostic model receiving unit to determine abnormal locations inside the sewer pipe from the unfolded image generated by the unfolded image generation unit.

The present invention also provides a server machine for a sewer pipe abnormality diagnosis support system that supports the diagnosis of abnormalities in sewer pipes using photographed images, which are video or still images, captured inside the sewer pipes, the server machine comprising: a teacher data storage unit that stores an unfolded image of the inside of the sewer pipe and abnormal locations in the unfolded image; a diagnostic model generation unit that uses the unfolded image and the abnormal locations in the unfolded image stored in the teacher data storage unit as teacher data and generates a diagnostic model by machine learning, in which the unfolded image is used as input and the abnormal locations in the unfolded image are used as output; and a diagnostic model transmission unit that transmits the diagnostic model generated by the diagnostic model generation unit.

The above server machine may be embodied as a cloud server.

The present invention also provides a method for a system for supporting diagnosis of abnormalities in sewer pipes, which supports diagnosis of abnormalities in sewer pipes using photographed images, which are videos or still images, taken inside the sewer pipes, the method including a photographed image capture unit capturing a photographed image of the inside of the sewer pipe, an unfolded image generating unit generating an unfolded image of the inside of the sewer pipe from the photographed image, an abnormality location determination unit determining an abnormal location inside the sewer pipe from the unfolded image, a determination result display unit displaying the locations determined to be abnormal by the abnormality location determination unit as determination results, and a change registration receiving unit accepting registration of changes to the determination result.

The present invention also provides a method by a client machine for a sewer pipe abnormality diagnosis support system that supports the diagnosis of abnormalities in a sewer pipe by using photographed images, which are video or still images, taken inside the sewer pipe, the method including: a photographed image capture unit captures a photographed image of the inside of the sewer pipe; an unfolded image generation unit generates an unfolded image of the inside of the sewer pipe from the photographed image; a client-side abnormality location determination unit determines abnormal locations inside the sewer pipe from the unfolded image; a determination result display unit displays the locations determined to be abnormal by the abnormality location determination unit as determination results; and a change registration acceptance unit accepts registration of changes to the determination result.

The present invention also provides a method by a server machine for a sewer pipe abnormality diagnosis support system that supports the diagnosis of abnormalities in sewer pipes using captured images, which are video or still images, captured inside the sewer pipes, the method including: a teacher data storage unit storing an unfolded image of the inside of the sewer pipe and abnormal locations in the unfolded image; a diagnostic model generation unit using the unfolded image and the abnormal locations in the unfolded image stored in the teacher data storage unit as teacher data to generate a diagnostic model by machine learning, in which the unfolded image is the input and the abnormal locations in the unfolded image are the output; and a diagnostic model transmission unit transmitting the diagnostic model generated by the diagnostic model generation unit.

According to the present invention, the presence or absence of abnormalities is determined from an unfolded view generated from still or moving images captured inside a sewer pipe, and changes to the determination results are registered, thereby improving diagnostic accuracy. In one example, the pipe type, pipe condition, etc. are determined from an unfolded view generated from still images captured inside a sewer pipe, and the presence or absence of abnormalities can be determined with greater accuracy by taking into account the determined pipe type and pipe condition information. In another example, the unfolded view and the determination results of the pipe type, pipe condition, etc. corresponding to the unfolded view and the analysis results of the abnormalities are accumulated in a storage device, and the correlations are repeatedly learned, thereby improving analysis accuracy.

1 is a conceptual diagram showing the configuration of a sewer pipe abnormality diagnosis support system according to an embodiment of the present invention; FIG. 1 is a diagram showing an example of a sewer pipeline. FIG. 2 is a block diagram showing the configuration of a client machine included in the sewer pipe abnormality diagnosis support system. FIG. 4 is a block diagram showing the configuration of an image analysis unit of a client machine. FIG. 4 is a block diagram showing a configuration of a client-side diagnostic model determining unit. FIG. 4 is a block diagram showing a configuration of a client-side pipe type estimation unit. FIG. 2 is a block diagram showing the configuration of a server machine, such as a cloud server, included in the sewer pipe abnormality diagnosis support system. 4 is a flowchart showing the operation flow of the sewer pipe abnormality diagnosis support system. FIG. 9 is a diagram for explaining the principle of creating a development view in the flow of FIG. 8 (generation of frame-by-frame image data). FIG. 9 is a diagram for explaining the principle of creating a development view in the flow of FIG. 8 (calculation of distance). FIG. 9 is a diagram for explaining the principle of creating a development view in the flow of FIG. 8 (an example of a distance meter). FIG. 9 is a diagram for explaining the principle of creating a development view in the flow of FIG. 8 (calculation of the cutting angle). FIG. 9 is a diagram for explaining the principle of creating a development view in the flow of FIG. 8 (cutting out an image based on a cut-out angle). FIG. 9 is a diagram for explaining the principle of creating a development view in the flow of FIG. 8 (principle of normal image conversion). FIG. 9 is a diagram for explaining the principle of creating a development in the flow of FIG. 8 (creation of a development by normal image transformation). FIG. 9 is a diagram for explaining the principle of creating a developed image in the flow of FIG. 8 (leveling the brightness of the developed image). FIG. 9 is a diagram for explaining the principle of development drawing generation in the flow of FIG. 8 (development drawing joining process). FIG. 13 is a diagram showing an example of a development obtained by creating a development. 9 is a flowchart showing the operation flow of image analysis in the flow of FIG. 8 . 9 is a flowchart showing the operational flow of analysis using a diagnostic model in the flow of FIG. 8 . 9 is a flowchart showing an operational flow of generating a diagnostic model in the flow of FIG. 8 . A diagram explaining the concept (learning stage) of random forest as an example of a learning algorithm. A diagram explaining the concept of random forest (operational stage) as an example of a learning algorithm. FIG. 1 is a diagram for explaining the concept of a neural network as an example of a learning algorithm. FIG. 9 is a diagram showing an example of a viewing screen for an abnormality diagnosis result in outputting an abnormality location in the flow of FIG. 8 . FIG. 9 is a diagram showing an example of a report (main management investigation record sheet) in the report output in the flow of FIG. 8 . An enlarged view of a portion of the main investigation record sheet in Figure 26. An enlarged view of a portion of the main investigation record sheet in Figure 26. An enlarged view of a portion of the main investigation record sheet in Figure 26 (an expanded image of a separate sheet). An enlarged view of a portion of the main investigation record sheet in Figure 26 (an expanded image of a separate sheet). An enlarged view of a portion of the main investigation record sheet in Figure 26 (an expanded image of a separate sheet). An enlarged view of a portion of the main investigation record sheet in Figure 26 (an expanded image of a separate sheet). 1 is a table of verification target data in an embodiment of a sewer pipe abnormality diagnosis support system. 5A to 5C are diagrams for explaining the principle of image analysis in the embodiment. 1 is a table showing the verification results of image analysis. 13 shows examples of abnormal locations to be verified in the embodiment (machine learning). FIG. 4 is a diagram showing a diagnostic model generation flow in the embodiment. 13 is a graph showing the results of determining abnormal (anomaly) locations using machine learning. 13 is a table showing the results of determining abnormal (anomaly) locations using machine learning.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, please note that the scope of the present invention is not limited to the following embodiment, but is defined by the description of the claims. For example, the sewer pipe abnormality diagnosis support system of the present invention is not limited to a server-client type system and may be configured by a single computer, or the functions of the client machine and the (cloud) server machine may be distributed to one or more arbitrary number of computers. It is not necessary for the sewer pipe abnormality diagnosis support system, client machine, and server of the present invention to have all of the functions described in the following embodiments. For example, it is not necessary to determine the investigation conditions such as the pipe type and the presence or absence of cleaning using a machine learning model, and the user may directly input them into the system. The machine learning algorithm for generating the machine learning model may be any algorithm, not limited to the random forest and neural network described below. For example, an image diagnosis model using deep learning can be used to estimate the investigation conditions such as the pipe type and the presence or absence of cleaning conditions, and to estimate the abnormality location. Each of the functional units described below may be realized by a single piece of hardware, or may be realized by two or more pieces of hardware, or a plurality of functional units may be realized by a single piece of hardware as described below. Note that in this specification, "water" may refer to pure water, or may refer to sewage such as polluted water, rainwater, or other water containing any impurities.

Fig. 1 is a conceptual diagram showing the configuration of a sewer pipe abnormality diagnosis support system according to one embodiment of the present invention, and Fig. 2 is a diagram showing an example of a sewer pipe. In one embodiment of the present invention, images (which may be videos or still images) of the inside of the sewer pipe 1000 shown in Fig. 2 are obtained by flying an unmanned aerial vehicle (drone) equipped with a camera to take pictures inside the sewer pipe 1000 shown in Fig. 2, or by driving a camera vehicle equipped with a video camera such as a television camera or a still image camera to take pictures, and the images are imported into a client machine, and abnormality location diagnosis is performed according to the following flow.
(1) Create an unfolded view from the captured image.
(2) Analyze pipe types, etc. from the development drawing (development image) using a pipe type, etc. analysis model. Also, retrain the pipe type, etc. analysis model using the result of final registration of changes, etc., made by the client to the analysis result.
(3) Using an abnormality analysis model, abnormalities such as damage and cracks are analyzed from the development diagram (developed image).
(4) The client registers the confirmed results of the abnormality location, including any changes, in response to the analysis results of the abnormality location, and retrains the abnormality location analysis model using the confirmed results.
(5) The diagnostic results are output as a report (in any format, such as a printed document or an electronic file).

FIG. 3 is a block diagram showing the configuration of a client machine included in the sewer pipe abnormality diagnosis support system, FIG. 4 is a block diagram showing the configuration of an image analysis unit of the client machine, FIG. 5 is a block diagram showing the configuration of a client-side diagnostic model determination unit, and FIG. 6 is a block diagram showing the configuration of a client-side pipe type estimation unit.

The client machine 1 includes a control unit 2, a storage unit 3, an input/output unit 4, and a communication unit 5. The control unit 2 includes a processor 6 such as a CPU (Central Processing Unit) and a temporary memory 7 such as a RAM (Random Access Memory). The processor 6 executes an unfolded image generation program 17 stored in the storage unit 3, causing the control unit 2 (the processor 6 of the control unit 2; the same applies below) to function as an unfolded image generation unit 8. The processor 6 executes an image analysis program 18 stored in the storage unit 3, causing the control unit 2 to function as an image analysis unit 10. The processor 6 executes a machine learning-related program 19 stored in the storage unit 3, causing the control unit 2 to function as a client-side diagnostic model determination unit 11 and a client-side pipe type estimation unit 12. Specifically, the processor 6 executes the diagnostic model receiving program, so that the control unit 2 functions as a diagnostic model receiving unit 32 using the communication unit 5; the processor 6 executes the abnormal part model determining program, so that the control unit 2 functions as an unfolded image classifying unit 33 and an abnormal part model determining unit 34; the processor 6 executes the pipe type estimation model receiving program, so that the control unit 2 functions as a pipe type estimation model receiving unit 35 using the communication unit 5; and the processor 6 executes the pipe type estimation program, so that the control unit 2 functions as a pipe type model estimating unit 36. The processor 6 executes the report output program 22 stored in the storage unit 3, so that the control unit 2 functions as a report output unit 15. In addition, the processor 6 executes various control and display programs, etc. 23 (including an operating system, various application software, driver software for various devices, etc.), so that the control unit 2 functions as a determination result display unit 13, a change registration accepting unit 14, and various control and display units 16. Any other program may be stored in the storage unit 3, and the control unit 2 can function as any functional unit by the processor 6 of the control unit 2 executing any program.

The unfolded image generating unit 8 is a functional unit that generates an unfolded image, for example according to the principle shown in Figures 9 to 18 described below, from photographed images of the inside of a sewer pipe (stored in the memory unit 3 as photographed image data 24) input from outside, for example via a data input/output unit of the input/output unit 4 (such as a Universal Serial Bus (USB) port). The unfolded image generating unit 8 stores the data of the generated unfolded image in the memory unit 3 as unfolded image data 25.

The image analysis unit 10 includes a pipe type and investigation condition reception unit 28, a binarization parameter setting unit 29, an image conversion unit 30, and a binary image analysis unit 31, and is a functional unit that identifies abnormal locations such as breakage and cracks by analyzing the unfolded image. The image analysis unit 10 stores data indicating the abnormal locations identified by the analysis (which may be data indicating the position or type of the abnormal location, or may be data of the unfolded image portion around the identified abnormal location) in the memory unit 3 as abnormal location and pipe type data 26.

The client-side diagnostic model determination unit 11 is a functional unit including the functional units of the diagnostic model receiving unit 32, the unfolded image classification unit 33, and the abnormality location model determination unit 34. The diagnostic model receiving unit 32 receives the machine-learned diagnostic model from the server machine 37 (see FIG. 7) described later and stores it in the storage unit 3 as the machine-learned diagnostic model 20. The unfolded image classification unit 33 classifies the unfolded image as necessary. The abnormality location model determination unit 34 analyzes the unfolded image using the machine-learned diagnostic model (the machine-learning algorithm may be any algorithm, and image recognition by deep learning may be used, or specific features may be extracted from the unfolded image and a model may be constructed by a random forest or neural network described later). The client-side diagnostic model determination unit 11 stores data indicating the abnormality location identified by the analysis (which may be data indicating the position and type of the abnormality location, or may be data of the unfolded image portion around the identified abnormality location) in the storage unit 3 as the abnormality location and pipe type data 26.

The client-side pipe type estimation unit 12 is a functional unit including the functional units of the pipe type estimation model receiving unit 35 and the pipe type model estimating unit 36. The pipe type estimation model receiving unit 35 receives the machine-learned pipe type estimation model from the server machine 37 and stores it in the memory unit 3 as the machine-learned pipe type estimation model 21. The pipe type model estimating unit 36 analyzes the unfolded image using the machine-learned pipe type estimation model (the machine learning algorithm may be any algorithm, and image recognition by deep learning may be used, or specific features may be extracted from the unfolded image and a model may be constructed by a random forest or neural network described later), thereby estimating the pipe type from the unfolded image. The client-side pipe type estimating unit 12 stores data indicating the pipe type identified by the analysis (and, if necessary, investigation conditions such as whether or not cleaning is performed) in the memory unit 3 as the abnormality location and pipe type data 26.

The judgment result display unit 13 is a functional unit that displays the parts judged to be abnormal by the client-side abnormal part judgment unit as the judgment result. In one example, a diagnosis result viewing screen with the layout shown in Figure 25 described later is displayed on the display device of the input/output unit 4.

The change registration reception unit 14 is a functional unit that accepts the registration of changes to the abnormality location determination result by the client-side abnormality location determination unit 9 via the keyboard and mouse of the input/output unit 4. The user of the client machine 1 can input confirmation registration, change registration, etc. of the abnormality location determination result by the client-side abnormality location determination unit 9 by operating the keyboard and mouse while checking the diagnosis result viewing screen as shown in FIG. 25, for example, and the change registration reception unit 14 stores the registration contents in the storage unit 3 as change registration data 27 (it may store only the change registration contents, or it may also store the abnormality location determination result by the client-side abnormality location determination unit as being "confirmed").

The report output unit 15 is a functional unit that outputs the results of the abnormality diagnosis of the sewer pipe as a report based on the judgment result by the client-side abnormality location judgment unit and the changes to the judgment result accepted for registration by the change registration acceptance unit 14. Examples of the report are shown in Figures 26 to 32 described below.

The storage unit 3 is a storage (recording) device equipped with a hard disk drive, SSD (Solid State Drive), etc., and stores the unfolded image generation program 17, the image analysis program 18, the machine learning related program 19 (including the diagnostic model receiving program, the pipe type estimation model receiving program, the abnormality location model determination program, and the pipe type estimation program), the report output program 22, and various control and display programs 23 as programs executed by the processor 6 to make the control unit 2 function as various functional units as described above. The storage unit 3 also stores a machine-learned diagnostic model 20 that determines an abnormality location from an unfolded image by image recognition, etc., as a machine-learned model generated by the diagnostic model generation unit 44 of the server machine 37. The storage unit 3 also stores a machine-learned pipe type estimation model 21 that estimates a pipe type from an unfolded image as a machine-learned model generated by the pipe type estimation model generation unit 46 of the server machine 37. The storage unit 3 also stores photographed image data 24, developed image data 25, abnormal location, pipe type data 26, and change registration data 27, as shown in FIG.

7 is a block diagram showing the configuration of a server machine, such as a cloud server, included in the sewer pipe abnormality diagnosis support system. The server machine 37 includes a control unit 38, a storage unit 39, an input/output unit 40, and a communication unit 41. The control unit 38 includes a processor 42, such as a CPU, and a temporary memory 43, such as a RAM. When the processor 42 executes a diagnostic model generation program from among the machine learning related programs 51 stored in the storage unit 39, the control unit 38 (the processor 42 of the control unit 38; the same applies below) functions as a diagnostic model generation unit 44, when the processor 42 executes a diagnostic model transmission program, the control unit 38 functions as a diagnostic model transmission unit 45 using the communication unit 41, when the processor 42 executes a pipe type estimation model generation program, the control unit 38 functions as a pipe type estimation model generation unit 46, when the processor 42 executes a pipe type estimation model transmission program, the control unit 38 functions as a pipe type estimation model transmission unit 47 using the communication unit 41, when the processor 42 executes a diagnostic model evaluation program, the control unit 38 functions as a diagnostic model evaluation unit 48, and when the processor 42 executes a pipe type estimation model evaluation program, the control unit 38 functions as a pipe type estimation model evaluation unit 49. In addition, when the processor 42 executes various control and display programs, etc. 54 (including an operating system, various application software, driver software for various devices, etc.), the control unit 38 functions as various control and display units 50. The memory unit 39 may store any other program, and the processor 42 of the control unit 38 executes the program, allowing the control unit 38 to function as any functional unit.

The diagnostic model generating unit 44 is a functional unit that generates a diagnostic model for estimating an abnormality from an unfolded image by machine learning. Any algorithm may be used as the machine learning algorithm, but in one example, a learning algorithm for image recognition by deep learning can be used in which the unfolded image data is input and information on the abnormality in the unfolded image data (position, type, and image data around the abnormality, etc.) is output. In machine learning using deep learning, a large number of sets of data of unfolded image data (input) and information on the abnormality in the unfolded image (position, type, and image data around the abnormality, etc. output) are prepared as teacher data (learning data), and the accuracy of the judgment of the model can be improved by having the image recognition model perform machine learning using this teacher data (in one example, the learning data is prepared as a large number of unfolded image data labeled with information on the abnormality). Image recognition by deep learning is widely known, and will not be described in further detail here. In addition, by preparing teacher data classified according to the investigation conditions such as pipe type and cleaning state, and generating separate diagnostic models according to the investigation conditions such as pipe type and cleaning state, it is possible to classify the unfolded image data by the investigation conditions such as pipe type and cleaning state, and then diagnose the abnormal part using the corresponding diagnostic model, which is considered to improve the judgment accuracy. The diagnostic model generation unit 44 stores the generated diagnostic model in the storage unit 39 as the machine-learned diagnostic model 52. The diagnostic model transmission unit 45 transmits the machine-learned diagnostic model generated by the diagnostic model generation unit 44 to the client machine 1 using the communication unit 41. The diagnostic model evaluation unit 48 evaluates the judgment accuracy of the machine-learned diagnostic model 52 using the unfolded image data and the abnormal part data as the test data 56, and evaluates the judgment accuracy of the machine-learned diagnostic model 52 by comparing the diagnosis result of the machine-learned diagnostic model 52 with the change contents on the client side indicated by the change registration data 57.

The pipe type estimation model generating unit 46 is a functional unit that generates a pipe type estimation model for estimating the pipe type from the unfolded image by machine learning. Any algorithm may be used as the machine learning algorithm, but in one example, a learning algorithm for image recognition by deep learning can be used in which the unfolded image data is input and the pipe type of the sewer pipe represented by the unfolded image (Hume pipe, ceramic pipe, PVC pipe, etc.) is output. In machine learning using deep learning, a large number of sets of data of unfolded image data (input) and the pipe type of the sewer pipe represented by the unfolded image (Hume pipe, ceramic pipe, PVC pipe, etc. output) are prepared as teacher data (learning data), and the image recognition model is made to perform machine learning using this teacher data (in one example, the learning data is prepared as a large number of unfolded image data labeled with information on the pipe type.) The accuracy of the model's judgment can be improved by having the image recognition model perform machine learning. In addition, a pipe type estimation model can be constructed as a model that estimates not only the pipe type but also the presence or absence of cleaning, etc. In this case, a large number of sets of data, each consisting of unfolded image data (input) and "sewer pipe type (Hume pipe, ceramic pipe, PVC pipe, etc.) and inspection conditions such as cleaning status" (output) represented by the unfolded image, are prepared as teacher data (learning data), and machine learning is performed on the image recognition model using this teacher data (in one example, learning data is prepared as a large number of unfolded image data labeled with information on inspection conditions such as pipe type and cleaning status (e.g., "Hume pipe, cleaned").) to improve the judgment accuracy of the model. The pipe type estimation model generation unit 46 stores the generated pipe type estimation model in the storage unit 39 as the machine-learned pipe type estimation model 53. The pipe type estimation model transmission unit 47 transmits the machine-learned pipe type estimation model generated by the pipe type estimation model generation unit 46 to the client machine 1 using the communication unit 41. The pipe type estimation model evaluation unit 49 evaluates the determination accuracy of the machine-learned pipe type estimation model 53 using the unfolded image data and pipe type data (which may include information on inspection conditions such as cleaning status) as test data 56, and evaluates the determination accuracy of the machine-learned pipe type estimation model 53 by comparing the diagnosis results of the machine-learned pipe type estimation model 53 with the changes on the client side indicated by the change registration data 57.

The storage unit 39 is a storage (recording) device equipped with a hard disk drive, SSD, etc., and stores various control and display programs 54, etc., as programs executed by the processor 42 to make the control unit 38 function as various functional units as described above. The storage unit 39 also stores a machine-learned diagnostic model 52 that determines an abnormality from an unfolded image by image recognition, etc., as a machine-learned model generated by the diagnostic model generation unit 44 of the server machine 37. The storage unit 39 also stores a machine-learned pipe type estimation model 53 that estimates a pipe type from an unfolded image as a machine-learned model generated by the pipe type estimation model generation unit 46 of the server machine 37. The storage unit 39 also stores teacher data 55, test data 56, and change registration data 57, as shown in FIG. 7.

Operation of the Sewer Pipe Abnormality Diagnosis Support System The operation of the sewer pipe abnormality diagnosis support system will be described below with reference to Figs. 8 to 32. Fig. 8 is a flowchart showing the operation flow of the sewer pipe abnormality diagnosis support system. Note that in the following example, the captured images will be described as moving images, but the sewer pipe abnormality diagnosis support system can be operated on the same principle even if still images are used (for example, image information similar to moving images can be obtained by intermittently capturing still images while a camera vehicle equipped with a still image camera drives inside a sewer pipe).

As already explained, it is assumed that video of the inside of the sewer pipe has been captured in advance by flying an unmanned aerial vehicle equipped with a camera inside the pipe to capture the video, and that the video (captured image) data 24 has been stored in the memory unit 3 of the client machine 1. The user of the client machine 1 selects the captured video to be read from the list of captured video data via the keyboard, mouse, etc. of the input/output unit 4 (the same applies to other inputs and outputs from the user) while referring to the screen displayed on the display device of the input/output unit 4 by the various control and display unit 16 (which also processes the browsing screen display as shown in FIG. 25 and the interface that accepts changes and registrations from the user). The various control and display unit 16 reads the selected captured video data from the captured video data 1 to N (N is an integer equal to or greater than 1) included in the capture data 24 and displays it on the display device of the input/output unit 4 (step S101).

Next, the user of the client machine 1 inputs instructions to create an unfolded diagram while referring to the screen displayed on the display device of the input/output unit 4 by the various control and display units 16, and the unfolded image creation unit 8 creates an unfolded diagram (unfolded image) using the selected captured video data (step S102).

Figures 9 to 17 are diagrams explaining the principle of creating a development view in the flow of Figure 8. However, the creation of the development view in this embodiment is not limited to the one shown in Figures 9 to 17, and any method may be used.

1. The unfolded image creation unit 8 first uses the functions of OpenCV (an open source library for computer vision) (included in the unfolded image generation program 17) to capture the video frame by frame and generate image data for each frame (Figure 9).

となる。画像中の左上の距離メータは、画像認識（パターンマッチング）で読み取る。また距離メータがない場合、下記「４．」以降の処理は、切り取り角度ｗｄｅｇ＝１８１．０として行う。 2. Next, the developed image generating unit 8 calculates the distance traveled (dist) from two consecutive images (FIG. 10). Here, the captured video includes a distance meter display (unit: m) as shown in FIG. 11. If the distance meter display in one image is A and the distance meter display in the next image is B, the distance traveled (dist) is calculated as follows:

The distance meter in the upper left of the image is read by image recognition (pattern matching). If there is no distance meter, the processing from "4." below onwards will be carried out with a cut angle wdeg = 181.0.

ただし「＊」は乗算を表す。

（２）ｒｌをｒｓ方向に縮めて行き、距離の合計が入力距離となるｒｓを算出

（３）上記で求めたｒｓから切り取り角度（ｗｄｅｇ）を算出

3. Next, the exfoliated picture generating unit 8 calculates the cutout angle (wdeg) of image A from the obtained distance (dist) in the following procedure (FIG. 12).
(1) Calculate the distance per pixel at point rl from the circumference of rl and the circumference calculated from the actual pipe diameter (input parameter).

Here, "*" represents multiplication.

(2) Calculate rs whose total distance is the input distance by shrinking rl in the rs direction.

(3) Calculate the cut angle (wdeg) from the rs obtained above.

（４）ｒｌの円周角度単位に正像変換で展開画像を作成。

※ｐａｒａｍ…可変パラメータ。別途チューニングして設定
上記角度パラメータより展開パラメータを計算する。

展開図幅単位に参照画素を求める。

4. Next, the developed picture generating unit 8 cuts out the image based on the cut angle in the following procedure (FIGS. 13 to 15).
(1) Recalculate rs from the cut angle wdeg. (2) The height of the development is the circumference pixel of rl + 1. (3) Calculate the width of the development using the following formula.

(4) Create an expanded image by normal transformation in units of the rl circular angle.

※param...variable parameter. Set by tuning separately. Calculate the expansion parameters from the above angle parameters.

Find the reference pixel per development width unit.

5. Next, the exfoliated image generating unit 8 averages the luminance of the exfoliated image in the following procedure (FIG. 16. Since there may be variations in the way the light hits the image, the luminance is averaged).
・Calculate the average luminance (lY) of the ceiling area.
If the difference between the average luminance (rY) of the target row and the luminance (Y) of the pixel being examined is equal to or less than ZZ (threshold value), the luminance is corrected.

6. Finally, the unfolded image generation unit 8 performs the joining process of the unfolded images (Figure 17).

Figure 18 is a schematic diagram showing an example of an unfolded view obtained by creating an unfolded view. In one example, the inner surface of a cylindrical sewer pipe is represented as a rectangular plan view as shown in Figure 18. Abnormal points 60 in the sewer pipe inner wall 58 are automatically detected by image analysis and judgment using a diagnostic model, which will be described later, and the detection results can be changed and registered on the client side. The unfolded image generated by the unfolded image generation unit 8 is displayed on the display device of the input/output unit 4.

Next, the user of client machine 1 selects either (A) image analysis or (B) analysis using a diagnostic model as the method of abnormality analysis for the displayed unfolded image (step S103).

(A) If image analysis is selected, the image analysis unit 10 performs image analysis of the abnormal area using the unfolded image generated in step S102 in the procedure shown in Figure 19 (step S104).

Figure 19 is a flowchart showing the operation flow of image analysis in the flow of Figure 8. Image analysis can be implemented using the OpenCV function (included in the image analysis program 18) in the same way as the generation of the unfolded image. First, the user manually sets the pipe type of the unfolded image and inputs it from the input/output unit 4, and the pipe type and investigation condition receiving unit 28 accepts the input (step S201). In one example, the user sets and inputs it from among a hume pipe, a ceramic pipe, and a PVC pipe. Next, the user manually sets the investigation conditions of the unfolded image and inputs it from the input/output unit 4, and the pipe type and investigation condition receiving unit 28 accepts the input (step S202). In one example, the user sets and inputs whether or not cleaning is performed. Next, the binarization parameter setting unit 29 generates parameters for image conversion (binarization) from the pipe type and investigation conditions (step S203). Specifically, the binarization parameter setting unit 29 sets a binarization threshold value that is predetermined for each pipe type and investigation condition (sets the threshold parameter of OpenCV). Next, the image conversion unit 30 performs binarization based on the parameters generated by "parameter generation" to convert the developed image into a black and white image (binary image) (step S204), and the binary image analysis unit 31 divides the image binarized by "image conversion" into pixel units (step S205). Next, the binary image analysis unit 31 scans the binary image pixel by pixel, marks parts containing black in black, and extracts them as black lumps (feature points) (step S206), and analyzes the extracted feature points to determine abnormal areas (step S207). Specifically, the binary image analysis unit 31 determines whether an abnormal area is an abnormal area using an algorithm that determines the characteristics of each abnormal area (breakage, cracks, etc.), and if an abnormal area is an abnormal area, determines the type of abnormality. As the algorithm here, any algorithm such as a publicly known object recognition algorithm by OpenCV or the like can be used, and the abnormal area is determined using information (parameters) on the connection, spread, and ratio of black areas that are predetermined for each type of abnormality. As a specific judgment criterion, for example, an abnormality is judged when the number of black pixels exceeds a predetermined threshold, and the black areas are classified as damaged, cracked, etc. depending on the shape of the black areas. Judgment conditions can be set and programmed based on the accumulation of know-how (patterns).

If analysis using diagnostic model (B) is selected in step S103, the client-side diagnostic model determination unit 11 performs image analysis of the abnormal area using the unfolded image generated in step S102 in the steps shown in Figures 20 and 21 (step S105).

First, the diagnostic model receiving unit 32 downloads the latest diagnostic model from the cloud server 37 and stores it in the storage unit 3 as the machine-learned diagnostic model 20 (step S301). Subsequently, the abnormality part model determining unit 34 uses the downloaded machine-learned diagnostic model 20 to determine the abnormality part (step S302). As described above, in one example, the machine-learned diagnostic model is a diagnostic model that has been machine-learned using a learning algorithm for image recognition by deep learning, in which the unfolded image data is input and information on the abnormality part in the unfolded image data (position, type, image data around the abnormality part, etc.) is output. The abnormality part model determining unit 34 uses the unfolded image data as input to the machine-learned diagnostic model 20 and determines the abnormality part by obtaining information on the abnormality part (position, type, image data around the abnormality part, etc.) as output.

As will be described in the following examples, the accuracy of the determination can be improved by clustering the unfolded image data using a method such as the k-means method beforehand and then performing a model determination of the abnormal area. In this case, the unfolded image classification unit 33 executes an abnormal area model determination program to classify the unfolded image data, and then the abnormal area model determination unit 34 determines the abnormal area using a diagnostic model 20 that has been machine-learned using an arbitrary algorithm. In one example, the unfolded image classification unit 33 first clusters the unfolded image data using the k-means method, and then the abnormal area model determination unit 34 extracts features from multiple parameters (background color, maximum brightness difference, range considered as background, etc.) from the unfolded image data according to a feature extraction method determined according to the clustering result, and inputs the extracted features as input to a diagnostic model machine-learned by a machine learning algorithm such as a random forest or neural network to perform a diagnosis, and information on the abnormal area (position, type, image data around the abnormal area, etc.) can be obtained as an output.

In step S105, the abnormality location may be determined in conjunction with the estimation of the investigation conditions such as the pipe type and cleaning state by the client-side pipe type estimation unit 12. In this case, in step S301, in addition to the above-mentioned processing, the pipe type estimation model receiving unit 35 downloads the latest pipe type estimation model from the cloud server 37 and stores it in the storage unit 3 as the machine-learned pipe type estimation model 21. Subsequently, in step S302, the pipe type model estimation unit 36 estimates the investigation conditions such as the pipe type and cleaning state using the downloaded machine-learned pipe type estimation model 21, and the abnormality location model determination unit 34 uses the machine-learned diagnostic model 20 corresponding to the estimation result of the pipe type model estimation unit 36 (in step S301, it is assumed that the diagnostic model receiving unit 32 receives the machine-learned diagnostic model 20 corresponding to the investigation conditions such as various pipe types and cleaning states and stores it in the storage unit 3).

In step S106, the judgment result display unit 13 outputs information on the abnormal location judged in step S104 or S105 (diagnosis result) as a system screen to the display device of the input/output unit 4. In one example, the diagnosis result viewing screen shown in FIG. 25 is displayed on the display device of the input/output unit 4 by the judgment result display unit 13. In the viewing screen of FIG. 25, a list of captured video data files is displayed in the upper left, and the user selects any captured video and instructs the above-mentioned development diagram generation and abnormal location diagnosis by pressing a button in the viewing screen. In addition, the development image is displayed at the bottom of the viewing screen, and in particular, a colored frame (displayed in grayscale in this drawing) is displayed to surround the location judged to be abnormal by image analysis or analysis using a diagnostic model. In addition, colored lines (shown in grayscale in the drawings) are displayed so as to overlap the connection parts (joint parts). (When generating a model for determining abnormal parts using an image analysis algorithm or machine learning, it can be configured to output not only information about the abnormal parts but also the positions of the connection parts (joint parts), etc. For example, when performing machine learning as deep learning on a diagnostic model, it is possible to provide the model with unfolded image data that is a set of information about the abnormal parts as well as information about the joint parts as training data to perform machine learning on the model, or an algorithm can be constructed as an image analysis algorithm so that when a linear part that looks like it is cutting a sewer pipe is detected in a binary image, it is determined to be a connection part (joint part).).

In addition, on the diagnosis result viewing screen displayed in step S106, the user can register (edit) changes to the abnormality determination results. In one example, when a user looking at the viewing screen of FIG. 25 finds an abnormality in a part not surrounded by the colored frame, the user can select the found abnormality by operating the mouse (drag-and-drop, etc.) on the input/output unit 4, and edit the abnormality determination results by selecting information such as the type of abnormality (breakage, cracks, etc.) according to the screen guidance. The user may also be able to edit the investigation conditions such as pipe type and cleaning state.

When editing is finished and the user has confirmed and changed the abnormality, the user presses the Complete button on the viewing screen, and the changes are stored in the memory unit 3 as change registration data 27, and the abnormality is confirmed (step S107). Even if there are no changes, the abnormality output by the user's operation is confirmed and the abnormality is confirmed. Next, the image data of the abnormality and the data of the abnormality after the user has changed it (which may include data after the investigation conditions such as pipe type and cleaning state have been changed) are sent from the communication unit 5 of the client machine 1 to the server machine 37 (abnormality data upload), and the communication unit 41 of the server machine 37 receives it, and the memory unit 39 stores the data of the abnormality after the change as change registration data 57 together with the image data of the abnormality (step S108). The diagnostic model generating unit 44 and the pipe type estimation model generating unit 46 of the server machine 37 use the received abnormality location data after the change by the user, the data after the change of the investigation conditions such as the pipe type and cleaning state, and the corresponding expanded image data (assumed to be received from the client machine 1) as teacher data to perform machine learning on the abnormality location diagnosis model and the pipe type estimation model again to generate machine-learned models, thereby improving the judgment accuracy of the abnormality location diagnosis model and the pipe type estimation model (step S109). Subsequently, the diagnostic model evaluating unit 48 and the pipe type estimation model evaluating unit 49 verify the accuracy of the generated machine-learned abnormality location diagnosis model and the pipe type estimation model by using the test data 56 or the like (step S110) and transmit them from the diagnostic model transmitting unit 45 and the pipe type estimation model transmitting unit 47 to the client machine 1, and the client machine 1 can use the abnormality location diagnosis model and the pipe type estimation model with improved judgment accuracy in the next analysis using the diagnostic model. In this way, by repeating the cycle of steps S105, S106, S107, S108, S109, S110, and S105, training data on abnormal locations is accumulated, and the diagnostic model generated by machine learning and the abnormality analysis and pipe type estimation accuracy of the pipe type estimation model are improved.

As already mentioned, the abnormality location diagnosis model and the pipe type estimation model may be machine-learned abnormality location diagnosis model and pipe type estimation model by deep learning, but are not limited to this and may be constructed by any machine learning algorithm such as random forest or neural network. An example of the operation flow for generating a diagnostic model is shown in FIG. 21.

In step S401, the diagnostic model generation unit 44 classifies images of abnormal areas by clustering. In one example, the diagnostic model generation unit 44 performs clustering by the k-means method. At this time, an arbitrary value N is set for the number of classifications, and the images are classified into N types. In step S402, the diagnostic model generation unit 44 generates parameters for creating a classifier for each classified image as follows:
[Parameter generation method]
Set the Open CV parameters (details below) determined for each classification type. <Vector file creation parameters>
-inv -randinv...Specify when inverting colors -bgcolor...Background color -bgthresh...Range to be considered as background -maxidev...Maximum brightness difference -maxxangle
-maxyangle... Maximum rotation angle rad
-maxangle
-w...vector width -h...vector height <classifier creation parameters>
- featureType ... feature finding method HAAR: A method to capture features from the brightness difference of an image LBP: A method to capture features from the brightness distribution (histogram) HOG: A method to capture features from the brightness gradient direction distribution - bt ... boost classifier type DAB: Discrete AdaBoost
RAB: Real AdaBoost
LB:LogitBoost
GAB:Gentle AdaBoost
-minHitRate: minimum hit rate -maxFalseAlarmRate: maximum false alarm rate -weightTrimRate: specifies whether trimming is performed by value -maxDepth: maximum depth of weak detector -maxWeakCount: maximum number of weak classifiers required to achieve maxFalseAlarmRate

In step S403, the diagnostic model generation unit 44 creates a classifier based on the parameters generated in the "classifier generation parameter generation" in step S402. The classifier here may be a classifier based on any machine learning algorithm, such as a random forest or a neural network.

Figure 22 is a diagram explaining the concept of random forest (learning stage) as an example of a learning algorithm, and Figure 23 is a diagram explaining the concept of random forest (operation stage). An overview of random forest will be explained below. In the operation stage, the final model machine-learned by random forest uses multiple models called decision trees, and the final output is obtained by taking the majority vote (classification) and average (regression) of the prediction (estimation) results by each decision tree. In the learning stage of random forest, a large number of explanatory variables are classified into multiple subsamples by random restoration extraction using a method called bootstrap method, and a large amount of teacher data in each subsample is given to each decision tree, so that each decision tree learns separately and independently, and learning is performed with multiple models (decision trees). The final machine learning model generated by the machine learning algorithm of random forest can be interpreted as a collection of multiple decision trees.

Figure 24 is a diagram explaining the concept of a neural network as an example of a learning algorithm. The overview of a neural network will be explained below. In a neural network, one or more hidden layers (intermediate layers) exist between the input layer and the output layer, and the node values in the input layer are converted to node values in the hidden layer, and the node values in the hidden layer are converted to node values in the output layer, thereby obtaining output data (objective variable data, classification results) from input data (explanatory variable data). The conversion of node values from one layer to the next layer is performed by linear conversion or nonlinear conversion using an activation function. The connections between the nodes between the input layer and the hidden layer, and the connections between the nodes between the hidden layer and the output layer each have a separate weight value, and the weight values are updated by providing teacher data of the explanatory variables and the objective variables and learning. During learning, the weights of each layer are updated by the backpropagation method. The difference between the required output and the actual output is calculated to be small, and reflected in each layer. A model can be constructed arbitrarily by adjusting hyperparameters such as the number of intermediate layers and the number of nodes belonging to each intermediate layer. Random forests and neural networks are well-known machine learning algorithms and will not be described in further detail here.

In step S404, the diagnostic model generation unit 44 generates a diagnostic model based on the classifiers created for each abnormality classification.

Once the abnormality has been identified in step S107, the report output unit 15 outputs the report (step S111). The report may be output on paper using a printer connected to the client machine 1, or in the form of an electronic file such as an EXCEL (registered trademark) file. Figure 26 shows an example of a report (main investigation record sheet) in the report output, and Figures 27 to 32 show enlarged views of a portion of the main investigation record sheet in Figure 26 (Figures 29 to 32 are expanded images of separate sheets). Note, however, that these do not represent actual abnormalities but are input into the system as examples.

Of the items on the form shown in Figures 27 and 28, survey number 61, survey name 62, upstream manhole number 63, downstream manhole number 64, (upstream) manhole type, manhole depth, soil overburden 65, pipe type, pipe diameter, route length 66, (downstream) manhole type, manhole depth, soil overburden 67, image of abnormal area 68, number of pipes, number of installed pipes, number of defective pipes, year of installation 69, distance 70, details of abnormal (broken) areas, distance 71, details of abnormal (crack) areas, distance 72, and number of each abnormality for each abnormal area 73 are items output from the system, and values registered in the system and their aggregated values are output. The other items are items prepared in advance as templates. In particular, the abnormality location image 68 outputs an image of the abnormality location registered in the system (an image of the abnormality location determined by image analysis or diagnostic model analysis and changed by the user as necessary), and the abnormality location details and distances 71 and 72 output information on the abnormality location registered in the system (anomaly type, rank, distance from the left end). Figures 29 to 32 show development images displayed on separate sheets, and in these, too, report output is performed based on registered information, such as the route number, upstream manhole number, downstream manhole number, investigation direction, number of pipes, extension, pipe diameter, pipe type, pipe length 74, damaged location 75, route number, upstream manhole number, downstream manhole number, investigation direction, number of pipes, extension, pipe diameter, pipe type, pipe length 76, and crack location 77.

Performance Evaluation of the Sewer Pipe Abnormality Diagnosis Support System The results of a performance evaluation of the sewer pipe abnormality diagnosis support system created as an embodiment of the present invention will now be described.

(System configuration of the embodiment)
In the system of this embodiment, the following specifications are installed in a TV camera vehicle:
・Traveling speed 21m/min ・Applicable pipe diameter 200-1200mm
Resolution: CMOS sensor Lens: Ultra-wide-angle lens (angle of view: 185°)
・Number of pixels: 5 million pixels・Other Images of sewer pipes taken with a camera that can be viewed in real time are imported into a personal computer (client machine) and an unfolded diagram is created. For the unfolded diagram, an abnormality is determined using a diagnostic model generated by image analysis and machine learning, and the parts mechanically determined to be abnormal are displayed on the unfolded diagram. The investigator (user) determines the abnormality such as a defect based on the results and creates an investigation report. The cloud-side function manages the investigator's final judgment result and the system judgment result (including false detection) as teacher data and generates a machine learning model (diagnostic model). This diagnostic model optimizes the machine learning model on the personal computer and is applied to the next and subsequent abnormality judgments. By repeating this cycle of determining abnormalities and improving the accuracy of the machine learning model, it is possible to improve the judgment accuracy and reduce the workload and investigation time required for diagnosis.

(Subject to inspection)
The accuracy of the system's judgment in this embodiment is verified by comparing the information on abnormal locations in the pipeline inspection results report (human diagnosis results) with the system's judgment results as the correct answer. The data to be verified were the results of screening surveys (pipe length 11.7 km) conducted in three cities, and the images of pipes with span-by-span abnormality judgment results of ranks A and B, mainly due to cracks and breakage, were sampled (see Table 1 in Figure 33).

(Identifying abnormal areas through image analysis)
Abnormal locations are identified by image analysis based on the differences between the abnormal locations and other image features. The process flow and characteristic processing image are shown in Figure 34. For the development diagram generated based on the screening investigation results, investigation conditions such as pipe type and whether cleaning is performed or not are set, and image processing parameters such as brightness and the necessity for noise removal are set according to each pipe type and the state inside the pipe. These parameters are values that make it easy to extract characteristic locations based on the conditions of the pipe investigation. Next, in the analysis process, the converted image is divided into pixel units, and these pixel units are marked in black. The marked black masses are extracted as characteristic points, and the abnormal locations and type of abnormality are identified based on their distribution (connection, spread, ratio, etc.).

(Verification results)
Anomaly location determination was performed on the verification target data (Table 1 in FIG. 33) using image analysis. The determination results showed that, among the abnormal locations of damage and cracks, ranks a and b determined that 23% to 35% were not abnormal, but all were able to be identified as abnormal. On the other hand, for small abnormalities of rank c, some locations could not be detected, and the abnormal location identification rate was 96.7% (Table 2 in FIG. 35). Although the determination by image analysis depends on the survey images such as the survey conditions, it is believed that, if the abnormality is relatively large, it will be possible to identify all abnormalities, even if some locations are not abnormal.

(Improved accuracy in detecting abnormalities using machine learning)
From the results of the verification so far, it has been confirmed that a certain level of accuracy can be secured in determining abnormal locations through image processing. However, when abnormalities that could not be extracted are detected, it is necessary to optimize parameters and improve the method for determining abnormal locations, and there are still issues to be addressed in order to permanently improve accuracy. Therefore, in order to permanently improve accuracy, it is considered useful to determine abnormal locations through machine learning, and in this study, although the amount of data is small, a machine learning model was generated using the diagnosis results as training data, and the accuracy of the determination was attempted to be verified.

(Creating diagnostic models using machine learning)
The accuracy of determining an abnormality of a model generated by machine learning depends on the amount of data used for learning. Therefore, in this verification, the accuracy of determination due to differences in the amount of learning data is compared for damage and cracks, which have a relatively large number of diagnostic results as teacher data. A plurality of diagnostic models are generated by changing the amount of learning data while keeping the ratio of abnormality to non-abnormality (7:3) the same. Anomaly determination other than the learning data was performed with each diagnostic model. In addition, since the learning data is small, the verification is performed for characteristic items of rank a (see FIG. 36), which are relatively easy to mechanically determine. The diagnostic model was generated using a generation flow using classifiers for damage and cracks (machine learning models generated as a result of learning (deep learning) for each image classified by clustering) (see FIG. 37). Images of abnormality are classified into images with common features by clustering using the k-means method. A diagnostic model for determining anomalies is generated by machine learning (deep learning) with parameters that extract the features.

(Verification results of abnormality detection using machine learning)
By increasing the amount of training data, the machine learning model was able to determine abnormalities in over 90% of cases, and about 70% of cases for cracks (Fig. 38, Fig. 39). This is thought to be because crack shapes have fewer characteristics than damage. From the results of this verification, we confirmed that a certain level of accuracy can be ensured for abnormalities that are relatively easy to identify even with a small amount of data, and that the accuracy of determination can be improved by increasing the amount of training data in the future.

The present invention can be used to diagnose abnormalities in various types of sewer pipes, including hume pipes, ceramic pipes, and PVC pipes.

1 Client machine 2 Control unit 3 Memory unit 4 Input/output unit 5 Communication unit 6 Processor 7 Temporary memory 8 Exploded image generation unit 9 Client-side abnormal part determination unit 10 Image analysis unit 11 Client-side diagnostic model determination unit 12 Client-side pipe type estimation unit 13 Determination result display unit 14 Change registration reception unit 15 Report output unit 16 Various control and display units 17 Exploded image generation program 18 Image analysis program 19 Machine learning related program 20 Machine learning completed diagnostic model 21 Machine learning completed pipe type estimation model 22 Report output program 23 Various control and display programs, etc. 24 Photographed video (image) data 25 Exploded image data 26 Abnormal part, pipe type data 27 Change registration data 28 Pipe type and investigation condition reception unit 29 Binarization parameter setting unit 30 Image conversion unit 31 Binarized image analysis unit 32 Diagnostic model reception unit 33 Exploded image classification unit 34 Abnormal part model determination unit 35 Pipe type estimation model reception unit 36 Pipe type model estimation unit 37 (Cloud) server machine 38 Control unit 39 Storage unit 40 Input/output unit 41 Communication unit 42 Processor 43 Temporary memory 44 Diagnostic model generation unit 45 Diagnostic model transmission unit 46 Pipe type estimation model generation unit 47 Pipe type estimation model transmission unit 48 Diagnostic model evaluation unit 49 Pipe type estimation model evaluation unit 50 Various control and display units 51 Machine learning related programs 52 Machine learning completed diagnostic model 53 Machine learning completed pipe type estimation model 54 Various control and display programs, etc. 55 Teacher data 56 Test data 57 Change registration data 58 Sewer pipe inner wall 59 Sewer 60 Abnormal (unusual) location 61 Investigation number 62 Investigation name 63 Upstream manhole number 64 Downstream manhole number 65 (Upstream) manhole type, manhole depth, soil overburden 66 Pipe type, pipe diameter, route length 67 (Downstream) manhole type, manhole depth, soil overburden 68 Abnormal location image 69 Number of pipes, number of installed pipes, number of defective pipes, year of installation 70 Distance 71 Details of abnormal (damaged) location, distance 72 Details of abnormal (crack) location, distance 73 Number of each abnormality for each abnormal location 74 Route number, upstream manhole number, downstream manhole number, investigation direction, number of pipes, extension, pipe diameter, pipe type, pipe length 75 Damaged location 76 Route number, upstream manhole number, downstream manhole number, investigation direction, number of pipes, extension, pipe diameter, pipe type, pipe length 77 Crack location 1000 Sewer pipe 1001 Ground surface 1002, 1003 Manhole 1100 Connection (joint part)
1200 Tubular space

Claims (16)

A sewer pipe abnormality diagnosis support system that supports abnormality diagnosis in a sewer pipe using a captured image, which is a video or a still image, captured in the sewer pipe, comprising:
an image capturing unit that captures an image of the inside of the sewer pipe;
an unfolded image generating unit that generates an unfolded image of the inside of the sewer pipe from the captured image;
an abnormality location determination unit that determines an abnormality location inside the sewer pipe from the unfolded image;
a judgment result display unit that displays the portion that is judged to be abnormal by the abnormal portion judgment unit as a judgment result to a user on a screen;
and a change registration receiving unit that receives registration of changes to the judgment result as input from the user based on the screen display to the user as the judgment result of the part judged to be abnormal by the abnormal part judgment unit . The abnormality location determination unit includes an image analysis unit,
a pipe type and inspection condition receiving unit that receives an input of a pipe type and inspection conditions;
a binarization parameter setting unit that sets binarization parameters according to the pipe type and the inspection conditions;
an image conversion unit that converts the exfoliated image into a binary image using the binarization parameters;
The system according to claim 1 , further comprising: a binary image analysis unit that analyzes the binary image to determine an abnormality. The abnormality location determination unit includes a diagnostic model determination unit,
a teacher data storage unit that stores an unfolded image of the inside of the sewer pipe and an abnormality location in the unfolded image;
a diagnostic model generating unit that uses the unfolded image stored in the training data storage unit and the abnormal portion in the unfolded image as training data to generate a diagnostic model by machine learning, the diagnostic model having the unfolded image as an input and the abnormal portion in the unfolded image as an output;
an abnormality location model determination unit that determines an abnormality location inside the sewer pipe from the unfolded image generated by the unfolded image generation unit by using the diagnostic model generated by the diagnostic model generation unit. The system according to claim 3, wherein the diagnostic model determination unit further includes an unfolded image classification unit that classifies the unfolded images by clustering prior to the abnormality location determination unit determining the abnormality location. The teacher data storage unit further stores a pipe type corresponding to the developed image of the inside of the sewer pipe,
a pipe type estimation model generating unit that uses the unfolded image stored in the teacher data storage unit and the pipe type corresponding to the unfolded image as teacher data to generate a pipe type estimation model by machine learning, the pipe type estimation model having the unfolded image as an input and the pipe type corresponding to the unfolded image as an output;
a pipe type estimation unit that estimates a pipe type corresponding to the unfolded image generated by the unfolded image generation unit, by using the pipe type estimation model generated by the pipe type estimation model generation unit. The system according to any one of claims 3 to 5, wherein the diagnostic model generation unit updates the diagnostic model by machine learning based on the changes to the judgment results that the change registration acceptance unit accepts to be registered. the captured image import unit, the developed image generation unit, the determination result display unit, and the change registration acceptance unit are embodied by a client machine;
Among the diagnostic model determination units included in the abnormality location determination unit, the teacher data storage unit and the diagnostic model generation unit are embodied by a server machine, and the abnormality location model determination unit is embodied by the client machine.
A system according to any one of claims 3 to 6. The system according to any one of claims 1 to 7, further comprising a report output unit that outputs the result of the abnormality diagnosis of the sewer pipe as a report based on the result of the abnormality determination unit and the changes to the determination result that have been accepted for registration by the change registration acceptance unit. A client machine for a sewer pipe abnormality diagnosis support system that supports abnormality diagnosis in a sewer pipe using a captured image, which is a video or still image, captured inside the sewer pipe, comprising:
an image capturing unit that captures an image of the inside of the sewer pipe;
an unfolded image generating unit that generates an unfolded image of the inside of the sewer pipe from the captured image;
a client-side abnormality determination unit that determines an abnormality in the inside of the sewer pipe from the unfolded image;
a determination result display unit that displays the portion determined to be abnormal by the client-side abnormal portion determination unit as a determination result to a user on a screen;
and a change registration receiving unit that receives registration of changes to the determination result as input content from the user based on the screen display to the user as a determination result of a part determined to be abnormal by the client-side abnormal part determination unit . The client-side abnormal portion determination unit includes an image analysis unit,
a pipe type and inspection condition receiving unit that receives an input of a pipe type and inspection conditions;
a binarization parameter setting unit that sets binarization parameters according to the pipe type and the inspection conditions;
an image conversion unit that converts the exfoliated image into a binary image using the binarization parameters;
The client machine according to claim 9 , further comprising: a binary image analysis unit that analyzes the binary image to determine an abnormal portion. The client-side abnormality location determination unit includes a client-side diagnostic model determination unit,
a diagnostic model receiving unit that receives a diagnostic model that is generated by machine learning using the unfolded image and an abnormal portion in the unfolded image as training data, and that takes the unfolded image as input and the abnormal portion in the unfolded image as output;
an abnormality part model determining unit that determines an abnormality part inside the sewer pipe from the unfolded image generated by the unfolded image generating unit by using the diagnostic model received by the diagnostic model receiving unit. A server machine for a sewer pipe abnormality diagnosis support system that supports abnormality diagnosis in a sewer pipe by using a captured image, which is a video or still image, captured in the sewer pipe,
a teacher data storage unit that stores an unfolded image of the inside of the sewer pipe and an abnormality location in the unfolded image;
a diagnostic model generating unit that uses the unfolded image stored in the training data storage unit and the abnormal portion in the unfolded image as training data to generate a diagnostic model by machine learning, the diagnostic model having the unfolded image as an input and the abnormal portion in the unfolded image as an output;
a diagnostic model transmission unit that transmits the diagnostic model generated by the diagnostic model generation unit;
and a change receiving unit that receives changes to the judgment result as input from a user based on a screen display of the judgment result of the abnormal portion using the diagnostic model. The server machine of claim 12, embodied as a cloud server. A method for a sewer pipe abnormality diagnosis support system that supports abnormality diagnosis of a sewer pipe by using a captured image, which is a video or a still image, captured inside the sewer pipe, comprising:
A photographed image capturing unit captures a photographed image of the inside of the sewer pipe;
an unfolded image generating unit generates an unfolded image of the inside of the sewer pipe from the captured image;
an abnormality location determination unit determines an abnormality location inside the sewer pipe from the unfolded image;
a determination result display unit displays on a screen to a user the portion determined to be abnormal by the abnormal portion determination unit as a determination result;
and a change registration accepting unit accepting registration of changes to the judgment result as input from the user based on the screen display to the user as a judgment result of the area judged to be abnormal by the abnormal area judgment unit . A method for a sewer pipe abnormality diagnosis support system using a client machine for supporting abnormality diagnosis of a sewer pipe by using a captured image, which is a moving image or a still image, captured inside the sewer pipe, comprising:
A photographed image capturing unit captures a photographed image of the inside of the sewer pipe;
an unfolded image generating unit generates an unfolded image of the inside of the sewer pipe from the captured image;
a client-side abnormality determination unit determines an abnormality in the inside of the sewer pipe from the unfolded image;
a determination result display unit displays on a screen to a user the portion determined to be abnormal by the abnormal portion determination unit as a determination result;
and a change registration accepting unit accepting registration of changes to the determination result as input from the user based on the screen display to the user as a determination result of a part determined to be abnormal by the client-side abnormal part determination unit . A method for a sewer pipe abnormality diagnosis support system, which supports abnormality diagnosis of a sewer pipe by using a captured image, which is a moving image or a still image, captured in the sewer pipe, comprising:
A teacher data storage unit stores an unfolded image of the inside of the sewer pipe and an abnormal portion in the unfolded image;
a diagnostic model generating unit uses the unfolded image stored in the training data storage unit and the abnormal portion in the unfolded image as training data, and generates a diagnostic model by machine learning, the diagnostic model having the unfolded image as an input and the abnormal portion in the unfolded image as an output;
a diagnostic model transmission unit transmitting the diagnostic model generated by the diagnostic model generation unit;
a change receiving unit receiving, as an input from a user, changes to the determination result based on a screen display of the determination result of the abnormality location using the diagnostic model.

Images