JP7679262B2 - Face evaluation support device and face evaluation support method - Google Patents

Description

The present invention relates to a face evaluation support device and a face evaluation support method.

To ensure the safety and ease of work in tunnel construction, it is necessary to properly evaluate the condition of the tunnel face. Traditionally, specialized engineers have visually inspected the face and recorded sketches of their observations, but in recent years, to address the shortage of specialized engineers and ensure the quality of the observation results, face evaluation devices have been developed that use image data (photographs) of the face and point cloud data that measures the three-dimensional shape of the face.

For example, Patent Document 1 discloses a rock weathering evaluation device that derives an evaluation category for weathering based on images of the rock face. Patent Document 2 discloses a crack evaluation device that evaluates cracks that develop in rock based on point cloud data that measures three-dimensional information on the tunnel face.

JP 2019-184541 A JP 2018-181271 A

In the devices disclosed in Patent Documents 1 and 2, image data and point cloud data are used to evaluate the face, respectively, so there is a limit to how much improvement can be made in the quality of the face evaluation by simply evaluating the image data and point cloud data separately. For example, the image data includes images of supports, roadbeds, etc. in addition to the face, so processing is required to separate the face area from areas other than the face on the image data, and if this processing is insufficient, it may reduce the quality of the face evaluation. In addition, the face may contain multiple rock masses with different characteristics, and if various observation items are evaluated without distinguishing between these rock masses, the condition of the face may not be evaluated appropriately.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide a face evaluation support device and face evaluation support method that make it possible to improve the evaluation quality when evaluating a face.

In order to achieve the above object, a face evaluation support device according to one aspect of the present invention comprises:
an image data acquisition unit that acquires image data of a target area including a tunnel face;
a point cloud data acquisition unit that acquires point cloud data obtained by measuring a three-dimensional shape of the target area;
A face evaluation unit that evaluates the face based on the image data and the point cloud data;
A support information output unit that outputs support information based on the evaluation result by the face evaluation unit,
The face evaluation unit is
A face extraction unit that extracts a face area corresponding to the face from the target area based on at least one of the image data and the point cloud data;
a roughness quantification unit that quantifies the roughness of the face in the face region as a roughness feature amount along a plurality of scanning lines set in a plurality of scanning directions with respect to the face region based on the point cloud data;
a region division unit that calculates a representative value of the roughness feature amount for each intersection point where a plurality of the scanning lines intersect, and divides the face region into divided regions based on a distribution of the representative value for each intersection point;
The apparatus is provided with a divided area evaluation processing unit which evaluates the face for each divided area into which the face area is divided by the area division unit, and outputs the evaluation result as the evaluation result.

According to a face evaluation support device according to one aspect of the present invention, a face extraction unit extracts a face region corresponding to the face based on at least one of image data and point cloud data, a roughness quantification unit quantifies the roughness of the face surface in the face region as a roughness feature based on the point cloud data, a region division unit divides the face region into divided regions based on the distribution of the roughness feature, and a divided region evaluation processing unit evaluates observation items related to the face for each divided region. Therefore, the region corresponding to the face is divided into divided regions, and observation items related to the face are evaluated for each divided region. Therefore, the evaluation quality can be improved when evaluating the face.

Other issues, configurations and advantages will be made clear in the detailed description of the invention described below.

1 is an overall view showing an example of a face evaluation support system 1. FIG. 1A and 1B show an example of data processed by the face evaluation support system 1, where (a) is image data and (b) is point cloud data. 2 is a block diagram showing an example of a face evaluation support device 4. FIG. FIG. 2 is a functional explanatory diagram showing an example of a face evaluation support device 4. 4 is a flowchart showing an example of a color correction process performed by a color correction unit 110. FIG. 2 is an explanatory diagram showing an example of a first face extraction process for extracting a face region 101 from point cloud data. An explanatory diagram showing an example of a second face extraction process for extracting a face area 101 from image data. 11 is an explanatory diagram showing an example of a roughness quantification process performed by a roughness quantification unit 131; FIG. 11 is an explanatory diagram showing an example of region partitioning processing by a region partitioning unit 132. FIG. An explanatory diagram showing an example of a crack dominant direction determination process by the crack dominant direction determination unit 133A. FIG. 11 is an explanatory diagram showing an example of rock group determination processing by the rock group determination unit 133B. An explanatory diagram showing an example of a crack spacing determination process by the crack spacing determination unit 133C. An explanatory diagram showing an example of a crack shape determination process by the crack shape determination unit 133D. 11 is an explanatory diagram showing an example of an inference process of an evaluation category for an observation item by the evaluation category inference unit 133E. FIG. 11 is an explanatory diagram showing an example of a continuity break determination process performed by a first continuity break detection unit 134A. FIG. 11 is an explanatory diagram showing an example of a continuity break determination process performed by a second continuity break detection unit 134B. FIG. 11 is an explanatory diagram showing an example of a continuity break determination process performed by a third continuity break detection unit 134C. FIG. FIG. 11 is an explanatory diagram showing an example of an artifact detection process performed by an artifact detection unit 134D. FIG. 11 is an explanatory diagram showing an example of a process for predicting areas at risk of skin falling off. A figure showing a face observation sheet 150, which is an example of a printed output of face observation sheet information 15. A figure showing a face sketch screen 160, which is an example of a screen display of face sketch information 16.

The following describes an embodiment of the present invention with reference to the drawings. The following shows a schematic view of the scope of the description necessary to achieve the object of the present invention, and mainly describes the scope of the description necessary to explain the relevant parts of the present invention. The parts that are omitted are based on publicly known technology.

Figure 1 is an overall view showing an example of a face evaluation support system 1. Figure 2 shows an example of data processed by the face evaluation support system 1, showing (a) image data and (b) point cloud data.

The tunnel face evaluation support system 1 is a system used to observe and evaluate the condition of the tunnel face F at the site. The main components of the tunnel face evaluation support system 1 include an image capture device 2, a three-dimensional shape measurement device 3, a tunnel face evaluation support device 4, a worker terminal device 5, and a manager terminal device 6. Each of the devices 2 to 6 is connected to a wired or wireless network 7 and is configured to be able to transmit and receive various data to and from each other. Note that there may be more than one of the devices 2 to 6, and the configuration of the network 7 is not limited to the example in FIG. 1.

The image capture device 2 is a device that captures an image of the target area 100 including the tunnel face F, and is composed of, for example, a digital camera with a color image sensor. The image capture device 2 is operated, for example, by a field worker, to capture an image of the target area 100 under predetermined capture conditions (capture position, installation position of the color sample 20, lighting, etc.), thereby generating color image data in which pixel values for the three components of RGB are recorded for each pixel. The color image data is sent to the face evaluation support device 4 via the network 7 or a recording medium, etc.

The image capture device 2 is used by being installed on the roadbed using a tripod or the like, but may also be attached to construction machinery or vehicles, or flying objects such as drones, unmanned aerial vehicles (UAVs), and balloons, or may be remotely operated. The image capture device 2 may also generate grayscale image data instead of color image data. In this embodiment, the image capture device 2 generates color image data and performs color correction using the color correction unit 110 described below. In the following, color image data will be abbreviated to "image data".

The image data includes not only the face F, but also the supports and roadbed. In addition, when the image data is captured, as shown in FIG. 2(a), a color sample 20 is placed near the front of the face F, so the image data also includes the color sample 20.

The color sample 20 has a predetermined black and white pattern made up of two colors, black and white, and is composed of, for example, three rectangular areas (each with approximately the same area) arranged in a black, white (with black borders on the top and bottom) and black row in the left-right direction. The color sample 20 adopts a point-symmetric or line-symmetric pattern so that the pattern remains the same even when the color sample is turned upside down. In addition, since the color sample 20 is only black and white, it can be easily made by, for example, printing the black pattern on blank copy paper or photographic paper (A3, B4, A4 size, etc.) using a printer (not compatible with color printing) and pasting it on a backing such as a blackboard, whiteboard, or cardboard.

The three-dimensional shape measuring device 3 is a device that measures the three-dimensional shape of the target area 100 including the tunnel face F, and is composed of, for example, a distance measuring sensor using a laser or ultrasound, a stereo camera, etc. The three-dimensional shape measuring device 3 measures the target area 100 under specified measurement conditions (measurement positions, etc.) to generate point cloud data that records the distance (depth) to each position in the target area 100 in the direction of the tunnel axis (z-axis). The point cloud data is sent to the face evaluation support device 4 via a network 7 or a recording medium, etc.

The three-dimensional shape measuring device 3 may use either an active method or a passive method for measuring a three-dimensional shape. The image capturing device 2 and the three-dimensional shape measuring device 3 may be configured as an integrated unit, and may be configured, for example, as an RGB-D (Red Green Blue Depth) camera capable of generating both image data and point cloud data.

As shown in FIG. 2(b), the point cloud data records the three-dimensional shape of the target area 100 based on the distance to the tunnel face F, with the tunnel axial direction being the z direction. In this case, there is no need to install a color sample 20. It is preferable that the shooting range of the image data and the measurement range of the point cloud data are approximately the same range, such as the target area 100, but they may be different ranges as long as both ranges correspond to the target area 100 that includes at least the entire tunnel face F. It is also preferable that the shooting time of the image data and the measurement time of the point cloud data are basically in the same time zone, but there may be a time difference as long as it is within a range in which the state of the tunnel face F does not change.

The face evaluation support device 4 is composed of a general-purpose or dedicated computer, and for example, a stationary computer or a portable computer is used. The face evaluation support device 4 acquires image data and point cloud data from the image capture device 2 and the three-dimensional shape measurement device 3, respectively, and performs various processes on the image data and point cloud data to evaluate various observation items related to the face F. The face evaluation support device 4 generates support information based on the evaluation results, and provides the support information to, for example, the worker terminal device 5, the manager terminal device 6, etc.

The face evaluation support device 4 also includes a face management database 400 that stores image data, point cloud data, and support information in association with each other. The face management database 400 stores a plurality of the above data related to the face F, and manages the data in an identifiable manner by adding attribute information such as an identifier (ID), site name, site location, and observation date and time.

The worker terminal device 5 is used, for example, by a worker on-site. The worker terminal device 5 is composed of a general-purpose or dedicated computer, and for example, a portable computer such as a tablet terminal or a smartphone is used. The worker terminal device 5 accepts various input operations and outputs various information via a display screen or voice using software such as an app or web browser. The worker terminal device 5 also transmits and receives various data to and from the face evaluation support device 4, etc., via the network 7.

The manager terminal device 6 is used, for example, by the on-site manager (which may be the same person as the worker). The manager terminal device 6 is composed of a general-purpose or dedicated computer, for example, a stationary computer or a portable computer. The manager terminal device 6 accepts various input operations and outputs various information via a display screen or voice using software such as an app or web browser. The manager terminal device 6 also transmits and receives various data to and from the face evaluation support device 4, etc., via the network 7.

Figure 3 is a block diagram showing an example of the face evaluation support device 4. The face evaluation support device 4 includes a storage unit 40 including a HDD, SSD, memory, etc., a control unit 41 including a processor such as a CPU, GPU, MPU, etc., an input unit 42 including a keyboard, mouse, etc., a display unit 43 including a display, etc., a communication unit 44 that is an interface with the network 7 based on a predetermined communication standard (which may be either wired or wireless), an external device interface (I/F) unit 45 that is an interface with external devices such as a printer, scanner, USB memory, etc., and a media input/output unit 46 that is an interface with storage media such as CDs and DVDs. The input unit 42 and the display unit 43 may be configured as a shared device such as a touch panel.

The memory unit 40 stores an operating system (OS), which is a basic program, a face evaluation support program 10 that controls the operation of the face evaluation support device 4, various data used by the face evaluation support program 10 (face management database 400, etc.), spreadsheet software that executes spreadsheet processing, document creation software that executes document creation processing, and web browser software, etc.

The face evaluation support program 10 and the various data used in the face evaluation support program 10 are basically stored in the memory unit 40, but these programs and data may be acquired from an external storage device via the communication unit 44 or the external device I/F unit 45, or may be acquired from a storage medium via the media input/output unit 46. In addition, it is preferable that the various data used in the face evaluation support program 10 be created in a format that can be displayed and edited using spreadsheet software, word processing software, and web browser software.

The control unit 41 executes the face evaluation support program 10 to function as an image data acquisition unit 11, a point cloud data acquisition unit 12, a face evaluation unit 13, and a support information output unit 14.

Figure 4 is a functional diagram showing an example of a face evaluation support device 4. The face evaluation support device 4 operates as the main body that implements the face evaluation support method, and the processing contents performed by each part of the face evaluation support device 4 correspond to each step of the face evaluation support method.

The image data acquisition unit 11 acquires image data of a target area 100 including the tunnel face F. The image data acquisition unit 11 acquires image data from the image capture device 2, for example, via the communication unit 44, the external device I/F unit 45, the media input/output unit 46, etc. The image data acquisition unit 11 also includes a color correction unit 110 that performs color correction on color image data. Note that the image data may be image data that was captured in the past and stored in the face management database 400, in which case the image data acquisition unit 11 can simply refer to the face management database 400.

The point cloud data acquisition unit 12 acquires point cloud data obtained by measuring the three-dimensional shape of the target area 100 including the tunnel face F. The point cloud data acquisition unit 12 acquires the point cloud data from the three-dimensional shape measurement device 3, for example, via the communication unit 44, the external device I/F unit 45, the media input/output unit 46, etc. Note that the point cloud data may be data that was previously measured and stored in the face management database 400, in which case the point cloud data acquisition unit 12 may simply refer to the face management database 400.

The face evaluation unit 13 evaluates the face F based on the image data acquired by the image data acquisition unit 11 and the point cloud data acquired by the point cloud data acquisition unit 12. The image data used by the face evaluation unit 13 is image data after color correction by the color correction unit 110.

The face evaluation unit 13 includes a face extraction unit 130, a roughness quantification unit 131, an area division unit 132, an area division evaluation processing unit 133, and a face area evaluation processing unit 134. Each unit will be described in detail later.

The section area evaluation processing unit 133 is configured to include a fracture dominant direction determination unit 133A, a rock group determination unit 133B, a fracture spacing determination unit 133C, a fracture shape determination unit 133D, and an evaluation section inference unit 133E. Any of the units 133A to 133E may be omitted.

The face area evaluation processing unit 134 is configured to include a first continuity crack detection unit 134A, a second continuity crack detection unit 134B, a third continuity crack detection unit 134C, an artificial object detection unit 134D, a first skin fall danger point prediction unit 134E, a second skin fall danger point prediction unit 134F, and a skin fall danger point evaluation unit 134G. Note that any of the units 134A to 134G may be omitted.

The support information output unit 14 outputs support information based on the evaluation result of the face F by the face evaluation unit 13. The support information is output in a format that can be handled by any software such as an app or a web browser. In addition, the output destination of the support information is, for example, the face management database 400, the worker terminal device 5, the manager terminal device 6, an external device (printer, USB memory), etc. In addition, the support information output unit 14 may output the support information in response to the image data and point cloud data being acquired by the image data acquisition unit 11 and the point cloud data acquisition unit 12, or may output the support information in response to a request from the worker terminal device 5, the manager terminal device 6, etc. In addition, the support information stored in the face management database 400 may be output in response to a request from, for example, the worker terminal device 5, the manager terminal device 6, etc.

The support information includes, for example, face observation sheet information 15, which records the evaluation results according to a specified format, and face sketch information 16, which displays the evaluation results superimposed on a display screen that displays image data.

Below, we will explain the details of each part of the face evaluation support device 4 (each process of the face evaluation support method) and the evaluation results of the face F by the face evaluation support device 4 with reference to Figures 5 to 21.

(1) Color correction processing by color correction unit 110 When the image data acquisition unit 11 acquires color image data in which the target area 100 including the face F and a color sample 20 having a predetermined black and white pattern is photographed in color as image data, the color correction unit 110 performs color correction on the color image data.

Figure 5 is a flowchart showing an example of color correction processing by the color correction unit 110.

First, in step S101, the color tone of the entire color image is corrected using the white and black pixels of the color sample 20 contained in the color image data.

Next, in step S102, the color image data is converted to a grayscale image, and the brightness of the pixel located in the middle when all the pixels that make up the grayscale image are arranged in order of brightness is calculated as the median brightness value.

Next, in step S103, all pixels are classified into a high-brightness pixel group consisting of pixels with a brightness higher than the median brightness value, and a low-brightness pixel group consisting of pixels with a brightness equal to or lower than the median brightness value.

Next, in step S104, the grayscale image is binarized using a threshold value of, for example, a minimum of 110 and a maximum of 253, and a white frame on the periphery of the color sample 20 is detected. Then, in order to remove noise, the grayscale image is expanded to join broken line segments, and a contour detection process is performed. From the contours obtained as a result, the position of the color sample 20 is detected based on the size, shape, area ratio, etc.

Next, in step S105, the average values of the three RGB components are calculated for the pixel group corresponding to the white portion of the color sample 20, and the maximum value of each average value is selected. Then, the difference values between the maximum value and each average value are determined as the correction value for the high-brightness pixel group. For example, if the average RGB values of the pixel group corresponding to the white portion are calculated as "230, 216, 231," "231" is selected as the maximum value, and the correction values for the high-brightness pixel group are determined as "1, 15, 0."

Next, in step S106, the high-brightness pixel group is color-corrected by adding the correction values for the high-brightness pixel group determined in step S105 to each pixel in the high-brightness pixel group for each of the three components. For example, if a specific pixel belonging to the high-brightness pixel group is "123, 234, 135," the pixel after correction using the above correction values will be "124, 249, 135."

Next, in step S107, the average values of the three components are calculated for the pixel group corresponding to the black portion of the color sample 20, and the minimum of the average values is selected. Then, the difference values between the minimum value and each average value are determined as the correction value for the low-brightness pixel group. For example, if the average values of RGB for the pixel group corresponding to the black portion are calculated as "24, 4, 6", "4" is selected as the minimum value, and the correction value for the low-brightness pixel group is determined as "20, 0, 2".

Next, in step S108, the low-luminance pixel group is color-corrected by subtracting the correction values for the low-luminance pixel group determined in step S107 from each pixel in the low-luminance pixel group for each of the three components. For example, if a specific pixel belonging to the low-luminance pixel group has values of "43, 30, 35," the pixel after correction using the above correction values will have values of "23, 30, 33."

As described above, the color correction unit 110 performs a series of color correction processes shown in FIG. 5 to correct color bias in the image data, so that the face F can be appropriately evaluated based on the corrected image data even when various conditions such as the type and illuminance of lighting, and the specifications and settings of the image capture device 2 are different.

(2) Face extraction process by face extraction unit 130 The face extraction unit 130 extracts a face region 101 corresponding to the face F from the target area 100 based on at least one of image data and point cloud data. Methods for extracting the face region 101 from the target area 100 include a first face extraction process for extracting the face region 101 by analyzing point cloud data, a second face extraction process for extracting the face region 101 from image data using a machine learning model, and a third face extraction process for using both the first face extraction process and the first face extraction process.

Figure 6 is an explanatory diagram showing an example of the first face extraction process for extracting the face area 101 from point cloud data. In the first face extraction process, when the point cloud data is measured in a coordinate system in which the tunnel axis is the z-axis (the coordinate value decreases the closer to the face side) as shown in Figure 6 (a), the face extraction unit 130 extracts the face area 101 based on the z-axis coordinate value of each point. For example, the advance distance L of the face face in one blast in tunnel construction is about 1.5 m with respect to the z-axis, and the unevenness of the face face with respect to the z-axis is less than the advance distance L. In addition, the supports are constructed at a position farther away from the face face than the advance distance L.

Therefore, as shown in FIG. 6(b), the face extraction unit 130 extracts the range from the minimum value of the z axis of the point cloud data acquired by the point cloud data acquisition unit 12 (the farthest point from the measurement position) to the travel distance L as the face region 101. Also, as shown in FIG. 6(c), the face extraction unit 130 extracts the face region 101 by performing a mask process on the image data acquired by the image data acquisition unit 11, assuming the range equal to or greater than the coordinate value obtained by adding the travel distance L to the minimum value of the z axis of the point cloud data as the region of the supports and roadbed.

As another face extraction process using point cloud data, the face extraction unit 130 may extract the face region 101 by utilizing the characteristic that, in the face region 101, the components along the z-axis are predominant in the normal vectors of the triangular network formed from the point cloud data, while in the support and roadbed regions, the components perpendicular to the z-axis components are predominant.

FIG. 7 is an explanatory diagram showing an example of the second face extraction process for extracting the face region 101 from image data. The machine learning model used in the second face extraction process is, for example, composed of a convolutional neural network (CNN), and machine learning is performed using training data including training image data of a face photographed and a correct answer label assigned to the face region 101 corresponding to the face F. That is, as shown in FIG. 7, in the learning phase of machine learning, the machine learning is performed by comparing the extraction result of the face region 101 output by inputting the training image data of the training data into the machine learning model with the correct answer label of the training data, and adjusting the weighting parameters of the convolutional neural network based on the comparison result. Note that the face extraction unit 130 may extract not only the face region 101, but also a support region corresponding to the support and a roadbed region corresponding to the roadbed portion, as shown in FIG. 7.

The face extraction unit 130 is the entity that executes the inference phase of machine learning, and extracts the face area 101 from the target area 100 by inputting the image data acquired by the image data acquisition unit 11 into a trained machine learning model.

In this manner, the face extraction process by the face extraction unit 130 extracts the face area 101 from the target area 100.

(3) Roughness quantification process by roughness quantification unit 131 The roughness quantification unit 131 quantifies, based on the point cloud data, the roughness (unevenness) of the face in the face region 101 as a roughness feature along a plurality of scanning lines set in a plurality of scanning directions for the face region 101 extracted by the face extraction unit 130. The roughness quantification unit 131 quantifies, as the roughness feature, the amplitude value and wavelength of a roughness approximation model obtained when the point cloud data corresponding to the face region 101 is approximated by a periodic function along the plurality of scanning lines.

Figure 8 is an explanatory diagram showing an example of roughness quantification processing by the roughness quantification unit 131. The roughness feature is a parameter that quantifies the roughness in the z-axis direction on the face based on the point cloud data. The roughness is approximated along multiple scanning lines i that are set in multiple scanning directions, for example, the up, down, left, right directions (x-axis, y-axis) and diagonal directions on the face as shown in Figure 8 (a), and are spaced at predetermined intervals d1 and d2.

The roughness approximated along the scanning line i is set from the z-axis coordinate value of the point cloud data located within a certain distance a from the scanning line i, as shown in Fig. 8(b). The height difference h _i (see Fig. 8(c)) from the reference line (surface) on the scanning line i is calculated by the following formula (1) using the coordinate value of the point group p _j included in the point cloud data and the distance r (see Fig. 8(b)) from the scanning line i to the point group p _j . Here, p _j,z is the z-axis coordinate value of the point group p _j , and z ₀ is the value of the reference line (surface).

The roughness of the face is approximated by a roughness approximation model for each scan line i, focusing on the magnitude of the height difference and the periodicity of the height difference. The roughness approximation model uses a periodic function to model (approximate) the height difference H(t) _i from the reference surface at position t of the scan line i by the following [Equation 2].

Here, the wave number _kj is a value equivalent to the reciprocal of the wavelength _λj , which is the repeat distance of the elevation difference. The coefficients _Aj and _Bj are feature quantities, and are obtained for the wave number _kj by, for example, using the least squares method or sparse modeling. The above roughness approximation model is a modeling that includes characterization by the average elevation difference by the superposition sum of j periodic functions. On the other hand, the wave number _kj may be set by using the crack interval (frequency), which is an observation item of the face F.

The roughness quantification unit 131 quantifies, for example, the amplitude value and wavelength in the roughness approximation model as the roughness feature amount for each scanning line i. The amplitude value _Cj of the roughness approximation model is calculated from coefficients _Aj and _Bj by the following formula [3]. Moreover, the wavelength _λj of the roughness approximation model is determined by the wave number _kj (=1/ _λj ) at which the amplitude value _Cj is maximum.

In this manner, the roughness of the face in the face region 101 is quantified as a roughness feature through the roughness quantification process performed by the roughness quantification unit 131.

(4) Area division processing by area division unit 132 The area division unit 132 calculates a representative value of the roughness feature quantified by the roughness quantification unit 131 for each intersection where multiple scanning lines intersect, and divides the face area 101 extracted by the face extraction unit 130 into divided areas 102 based on the distribution of the representative values for each intersection.

9 is an explanatory diagram showing an example of region division processing by the region division unit 132. When dividing the face region 101 into the divided regions 102, the region division unit 132 obtains a representative value of the roughness feature amount at each intersection P where a plurality of scanning lines intersect. For example, as shown in FIG. 9A, when the amplitude value C and the wavelength λ are quantified as the roughness feature amount at the intersection P _{i, j, k, l} where four scanning lines in the vertical and diagonal directions intersect, the representative value of the roughness feature amount is obtained by the average (simple average or weighted average) or vector sum of the roughness feature amount of the four scanning lines, i.e., the amplitude value C and the wavelength λ.

9B, the region partitioning unit 132 compares the representative values (amplitude value C and wavelength λ) for each intersection P with the reference value C ₀ of the amplitude value C and the reference value λ ₀ of the wavelength λ to determine which of the four characteristic partitions Ra, Rb, Rc, or Rd each intersection P is classified into. Note that the characteristic partitions may be classified into other characteristic partitions by setting multiple reference values for the amplitude value C or multiple reference values for the wavelength λ.

Then, the region division unit 132 divides the face region 101 into divided regions 102 based on the distribution of the characteristic divisions Ra, Rb, Rc, and Rd into which each intersection P is classified. FIG. 9(c) illustrates an example in which the face region 101 is divided into two divided regions 102: a divided region 102 (upper left part) which is a collection of intersections P classified into characteristic division Ra, and a divided region 102 (lower right part) which is a collection of intersections P classified into characteristic division Rd.

In this manner, the area division process by the area division unit 132 divides the face area 101 into divided areas 102 based on the distribution of roughness features.

(5) Sectional Area Evaluation Processing by Sectional Area Evaluation Processing Unit 133 The sectional area evaluation processing unit 133 evaluates the observation items related to the face F as evaluation results for each sectional area 102 into which the face area 101 is divided by the area dividing unit 132. In order to evaluate various observation items for each sectional area 102 based on the image data and point cloud data corresponding to the sectional area 102, the sectional area evaluation processing unit 133 includes a fracture dominant direction determination unit 133A, a rock group determination unit 133B, a fracture spacing determination unit 133C, a fracture form determination unit 133D, and an evaluation section inference unit 133E, as shown in FIG. 4. Each unit (each process) 133A to 133E will be described in detail below.

(5-1) Crack dominant direction determination unit 133A
10 is an explanatory diagram showing an example of a process of determining a predominant crack direction by the predominant crack direction determining unit 133A. The predominant crack direction determining unit 133A polar projects each normal vector of a triangular mesh composed of point cloud data corresponding to the face region 101 onto a stereo net for each divided region 102, and determines the predominant crack direction indicating the strike and dip of the predominant crack on the face for each divided region 102 based on the distribution of the polar projected plots on the stereo net.

At the face, rock masses are often separated and excavated along existing cracks, so unevenness on the face is likely to appear along the cracks. Therefore, in order to determine the predominant crack direction in each divided area 102, the predominant crack direction determination unit 133A uses a triangular mesh composed of point cloud data corresponding to the divided area 102 as shown in FIG. 10(a), and polar projects the normal vectors of each triangle included in the triangular mesh onto a stereonet (equal area projection net) as shown in FIG. 10(b) and (c). In polar projection onto the stereonet, the arrow base end of each normal vector is aligned with the center of the stereonet, and each is projected in a direction that coincides with the tunnel axis (z-axis).

Then, as shown in FIG. 10(c), the crack dominant direction determination unit 133A creates a density contour diagram based on the distribution of plots corresponding to the arrow tip side of each normal vector polar projected onto the stereo net, and determines the range in the density contour diagram where the density is higher than a predetermined reference value as the crack dominant direction. In this case, the crack dominant direction determination unit 133A determines the crack dominant direction for each partitioned area, but there may be cases where it is determined that one partitioned area contains multiple crack dominant directions.

When the point cloud data is irregularly distributed on the face, it is expected that the number of plots polar projected onto the stereonet will not correspond to the area of the triangles on the face. In that case, the crack dominant direction determination unit 133A may weight the triangles that make up the triangular network by the area of the triangles and plot them on the stereonet, thereby determining the crack dominant direction while reflecting the area on the face.

(5-2) Rock group determination unit 133B
11 is an explanatory diagram showing an example of a rock group determination process by the rock group determination unit 133B. The rock group determination unit 133B determines a rock group indicating the type of rock constituting the face for each divided area 102 based on the amplitude value and wavelength of the roughness approximation model quantified by the roughness quantification unit 131.

Rock groups are classified into a number of rock types, for example, by combining weathering classifications of either massive or layered, and strength classifications of either hard, medium-hard, or soft. For example, in rock group Ga, which is hard rock/massive (e.g., granite, etc.), the amplitude value indicating the unevenness of the cracks tends to be large, and the wavelength indicating the spacing of the cracks tends to be medium to long. In rock group Ge, which is soft rock/layered (e.g., Tertiary shale, phyllite, etc.), fine bedding and schistosity are present, so the amplitude value becomes small and the wavelength becomes small due to the unevenness along the bedding and schistosity. In rock group Gc, which is soft rock/massive (e.g., tuff breccia with low degree of consolidation), the face is smoothly finished by excavation, so there is almost no unevenness, the amplitude value becomes small, and the wavelength becomes long. The rock group determination unit 133B determines the rock group of each divided area 102 by focusing on the tendencies and characteristics of each rock group as described above, for example, based on the amplitude value and wavelength at each intersection point P. Note that the rock group determination unit 133B may determine the rock group of each divided area 102 by referring not only to the roughness feature amount but also to the evaluation classification for other observation items (e.g., weathering, compressive strength, etc.).

(5-3) Crack Spacing Determination Unit 133C
12 is an explanatory diagram showing an example of a process of determining the crack spacing by the crack spacing determiner 133C. The crack spacing determiner 133C performs an edge detection process to detect edges in image data corresponding to the partitioned region 102, and determines the crack spacing for each partitioned region 102 based on the edge density of the edges detected by the edge detection process by referring to correspondence information that defines the relationship between the edge density and the crack spacing.

The crack spacing determination unit 133C detects edges from the image data using any edge detection process such as the Canny method, and calculates the edge density. The crack spacing determination unit 133C then determines the relationship between edge density and crack spacing by, for example, referring to correspondence information defined for each rock type, and identifying the crack spacing associated with the edge density calculated as described above in the correspondence information for the rock group determined by the rock group determination unit 133B.

In the above description, multiple pieces of correspondence information are defined for each type of rock, but this is not limited to the above. For example, multiple pieces of correspondence information may be defined for each other observation item, or a single piece of correspondence information may be defined.

(5-4) Crack shape determination unit 133D
13 is an explanatory diagram showing an example of a process of determining a crack form by the crack form determining unit 133D. The crack form determining unit 133D determines the crack form for each divided area 102 based on the wavelength characteristics according to the scanning direction of the scanning line when the roughness quantification unit 131 approximates the roughness feature amount to a roughness approximation model.

The roughness feature quantity quantified along the scan line by the roughness quantification unit 131 can be used to estimate the crack shape. When the wavelength λ, which is the distance of repeated height differences in scan lines set in the up, down, left, right, and diagonal directions, is roughly constant regardless of the direction of the scan line (within a range of a predetermined reference value), the crack shape determination unit 133D determines that the crack shape of the partitioned area is "random square" or "sand-like." On the other hand, when the wavelength λ varies greatly depending on the direction of the scan line (outside the range of a predetermined reference value), the crack shape determination unit 133D determines that the crack shape of the partitioned area is "columnar" or "layered."

(5-5) Evaluation category inference unit 133E
14 is an explanatory diagram showing an example of an inference process of an evaluation category related to an observation item by the evaluation category inference unit 133E. The evaluation category inference unit 133E infers an evaluation category for each partitioned area 102 by inputting image data corresponding to the partitioned area 102 into a machine learning model. The machine learning model used by the evaluation category inference unit 133E is obtained by machine learning using training data including learning image data of an image of a face F and a correct answer label in which an evaluation category related to an observation item is assigned to the face F.

Examples of observation items to be inferred include compressive strength, weathering, crack spacing, crack morphology, crack condition, and rock group. The evaluation category inference unit 133E may infer an evaluation category for at least one of these observation items. Note that the observation items of crack spacing, crack morphology, crack condition, and rock group can be determined by the above-mentioned crack spacing determination unit 133C, crack morphology determination unit 133D, and rock group determination unit 133B, but the evaluation category inference unit 133E may infer each observation item instead of these, or may infer each observation item in combination with these.

The machine learning model is preferably constructed, for example, by a convolutional neural network, and is created for each observation item, as shown in FIG. 14. For example, the machine learning model for compressive strength is machine-learned using training image data of a face photographed and teacher data including a correct answer label to which an evaluation category for compressive strength is assigned for the face F. The evaluation category inference unit 133E infers an evaluation category for compressive strength for a partitioned area by inputting image data corresponding to the partitioned area into a trained machine learning model. At that time, the image data may be input to the machine learning model in a state in which the image data is divided into images of small areas having a predetermined size, and in that case, the evaluation category inference unit 133E may infer an evaluation category for compressive strength for each small area included in the partitioned area. The machine learning models for other observation items are constructed in the same manner as above, and therefore will not be described.

(6) Face area evaluation processing by the face area evaluation processing unit 134 The face area evaluation processing unit 134 evaluates the observation items related to the face F as evaluation results for the face area 101 extracted by the face extraction unit 130. In order to evaluate various observation items, the face area evaluation processing unit 134 includes a first continuity crack detection unit 134A, a second continuity crack detection unit 134B, a third continuity crack detection unit 134C, an artificial object detection unit 134D, a first skin fall danger location prediction unit 134E, a second skin fall danger location prediction unit 134F, and a skin fall danger location evaluation unit 134G, as shown in FIG. The details of each unit (each process) 134A to 133G will be described below.

(6-1) First continuity break detection unit 134A
15 is an explanatory diagram showing an example of the continuity crack determination process by the first continuity crack detection unit 134A. The first continuity crack detection unit detects continuous cracks formed continuously on the face based on point cloud data. Specifically, the first continuity crack detection unit extracts candidates for edges that form part of the continuity crack based on the angle formed by the normal vectors of two triangles including a common side from each side of a triangular net composed of point cloud data corresponding to the face region 101, and detects the continuity cracks present in the face region 101 by determining the continuity between the candidates.

In point cloud data, the uneven shape of the face is represented by a triangular mesh, but the sides of each triangle are a mixture of those that represent the edges of the unevenness of the face and those that represent convenient dividing lines. Therefore, continuous cracks are identified by extracting the sides of the triangles that represent the edges of the unevenness of the face and considering the continuity of each extracted side.

The determination of whether side ab common to two adjacent triangles A and B represents a convex/concave edge (edge determination process) is performed based on the angle between normal vectors nA and nB of triangles A and B, as shown in FIG. 15. For example, if the angle θ between the two normal vectors nA and nB is within a predetermined threshold range of 0° or 180°, triangles A and B are considered to be on the same plane, and side ab is determined to not be a convex/concave edge; otherwise, side ab is determined to be a convex/concave edge. In this case, the angle θ between the normal vectors nA and nB may be calculated, or the inner product of the normal vectors nA and nB may be used.

The first continuity crack detection unit performs the above edge determination process on all edges shared by two adjacent triangles, and extracts edge candidates that are part of the continuity crack. Then, for each of the extracted edge candidates, it determines whether there is continuity between the edge candidates (continuity determination process), and groups the edges by connecting consecutive edges as broken lines. In the continuity determination process, the two edges to be determined are treated as line segment vectors, and if the angle between the line segment vectors of the two edges is equal to or less than a predetermined threshold, it is determined that the edges are consecutive, and if not, it is determined that the edges are separated. The first continuity crack detection unit detects continuity cracks by deleting broken lines that are shorter than a predetermined threshold from the grouped broken lines, and further selecting those that are more straight.

(6-2) Second continuity break detection unit 134B
FIG. 16 is an explanatory diagram showing an example of a process for determining a continuity crack by the second continuity crack detection unit 134B. The second continuity crack detection unit detects continuous cracks formed continuously on the face surface based on image data. Specifically, the second continuity crack detection unit detects continuous cracks present in the face region 101 by inputting image data corresponding to the face region 101 into a machine learning model. The machine learning model used by the second continuity crack detection unit is, for example, composed of a convolutional neural network, and machine learning is performed using teacher data including learning image data of a face photographed and a correct answer label assigned to the continuous crack formed in the face F.

(6-3) Third continuity break detection unit 134C
17 is an explanatory diagram showing an example of the determination process of the continuity crack by the third continuity crack detection unit 134C. The third continuity crack detection unit detects the continuity cracks formed continuously on the face based on the image data and the point cloud data. Specifically, the third continuity crack detection unit performs an edge detection process to detect edges on the image data corresponding to the face region 101, and detects the continuity cracks present in the face region 101 by fitting the edges detected by the edge detection process to the crack dominant direction for each divided region 102 determined by the crack dominant direction determination unit 133A for the point cloud data.

The third continuity crack detection unit detects edges from the image data using any edge detection process, such as the Canny method. The third continuity crack detection unit then detects continuity cracks by performing a fitting process that fits the detected edges to the crack dominant direction for each divided area 102, for example, using the least squares method or sparse modeling.

(6-4) Artifact detection unit 134D
FIG. 18 is an explanatory diagram showing an example of a detection process of an artificial object by the artificial object detection unit 134D. The artificial object detection unit 134D detects an artificial object present in the target area 100 based on image data corresponding to the target area 100. Specifically, the artificial object detection unit 134D separates the image data corresponding to the target area 100 into image data corresponding to the face area 101 and image data corresponding to areas other than the face area 101 (for example, a support area or a roadbed area), and inputs the image data into a machine learning model, thereby detecting an artificial object present in the face area 101 and an artificial object present outside the face area 101. The machine learning model used by the artificial object detection unit 134D is obtained by machine learning using training data including learning image data of a face photographed and a correct answer label assigned to an artificial object installed at the face or around it.

Examples of man-made objects to be detected include steel supports, rock bolts, head bolts, pre-work, blackboards, people, and shadows of people and machinery. The man-made object detection unit 134D may detect at least one of these man-made objects. Man-made objects to be detected in the face region 101 include, for example, head bolts, pre-work, blackboards, people, and shadows of people and machinery, while man-made objects to be detected in the support region and roadbed region other than the face region 101 include, for example, rock bolts, steel supports, blackboards, people, and shadows of people and machinery, but are not limited to these examples. FIG. 18 illustrates an example in which the man-made object detection unit 134D detects multiple pre-work in the face region 101 and multiple rock bolts in the support region.

The machine learning model is, for example, configured with a convolutional neural network, and the machine learning model for detecting artifacts from image data corresponding to the face region 101 and the machine learning model for detecting artifacts from image data corresponding to regions other than the face region 101 may be created as a common machine learning model, or may be created as separate machine learning models. Furthermore, a machine learning model may be created for each type of artifact to be detected. Furthermore, the image data may be input to the machine learning model in a state in which the image data is divided into images of small regions having a predetermined size, and in that case, the artifact detection unit 134D may detect artifacts in units of small regions included in the target region 100.

(6-5) First skin loss risk location prediction unit 134E
19 is an explanatory diagram showing an example of a process for predicting a skin-fall danger area. The first skin-fall danger area prediction unit 134E predicts a skin-fall danger area indicating an area where a risk of skin-fall exists at the face F based on point cloud data. Specifically, as shown in FIG. 19, the first skin-fall danger area prediction unit 134E projects the crack dominant direction for each divided area 102 determined by the crack dominant direction determination unit 133A onto a stereo net as a great circle, and when a closed polygon is formed by the crack dominant directions extending in at least three directions, predicts the range of the face corresponding to the closed polygon as a skin-fall danger area.

The first skin-fall danger area prediction unit 134E predicts skin-fall danger areas by geometrically utilizing the predominant crack direction for each divided area 102 determined by the predominant crack direction determination unit 133A for the point cloud data. For example, if the predominant crack direction is projected as a great circle onto a stereonet and a spherical triangle is formed on the stereonet, the area corresponding to the spherical triangle is predicted to be a block that can escape from the face.

(6-6) Second skin loss risk location prediction unit 134F
The second skin-fall danger area prediction unit 134F predicts, based on image data, skin-fall danger areas that indicate areas where there is a risk of skin-fall at the face F. Specifically, as shown in Fig. 19, the second skin-fall danger area prediction unit 134F predicts skin-fall danger areas that exist in the face area 101 by inputting image data corresponding to the face area 101 into a machine learning model that has been machine-learned using training data that includes a photograph of the face and a correct answer label that has been assigned to the skin-fall danger areas at the face F where there is a risk of skin-fall.

The machine learning model is, for example, configured with a convolutional neural network. The image data may be divided into small area images of a predetermined size and input to the machine learning model. In this case, the second skin-fall risk area prediction unit 134F may infer whether or not there is a risk of skin-fall for each small area included in the face area 101.

(6-7) Skin fall risk location evaluation unit 134G
As shown in FIG. 19, the skin fall danger area evaluation unit 134G predicts skin fall danger areas based on the prediction results by the first skin fall danger area prediction unit 134E and the prediction results by the second skin fall danger area prediction unit 134F. Specifically, the skin fall danger area evaluation unit 134G compares the skin fall danger areas predicted by the first skin fall danger area prediction unit 134E with the skin fall danger areas predicted by the second skin fall danger area prediction unit 134F, and determines that skin fall danger areas with a high degree of agreement as a result of the comparison are at high risk.

The skin-fall danger point evaluation unit 134G may predict the skin-fall danger points taking into account other observation items. For example, the skin-fall danger point evaluation unit 134G may determine that there is a high risk of skin falling when, for example, a mirror bolt is detected as an artificial object detected by the artificial object detection unit 134D. Furthermore, when either the first skin-fall danger point prediction unit 134E or the second skin-fall danger point prediction unit 134F is omitted, the skin-fall danger point evaluation unit 134G may output the skin-fall danger points predicted by the other skin-fall danger point evaluation unit as is, or the skin-fall danger point evaluation unit 134G itself may be omitted.

(7) Support information output processing by the support information output unit 14 As described above, the support information output unit 14 generates face observation sheet information 15, face sketch information 16, etc. as support information based on the evaluation results of the face F for each observation item by the face evaluation unit 13, and outputs it in an output form such as data storage output, screen display output, print output, etc.

Figure 20 shows a face observation sheet 150, which is a printout example of face observation sheet information 15. The face observation sheet 150 records the evaluation results of the face F for each observation item in a specified format. In the example of Figure 20, the face observation sheet 150 includes a face information column 151 in which information about the face F is recorded, and an evaluation category column 152 in which the evaluation results of the face F are recorded.

In the face information column 151, for example, the site name, site location, observation date and time, etc. are recorded. In the evaluation category column 152, for example, the evaluation category for compressive strength, weathering, crack spacing, crack condition, strike and dip (crack dominance direction), etc. are recorded.

In the evaluation category column 152 shown in FIG. 20, the evaluation category for each area when the face F is divided into three areas, the top, left shoulder, and right shoulder, is recorded. This is the evaluation result evaluated for each observation item for each divided area 102 by the face evaluation unit 13 (particularly the divided area evaluation processing unit 133) is converted into an evaluation result for the top, left shoulder, and right shoulder by weighting and adding the evaluation result for each divided area 102 based on the area ratio of overlap between each divided area 102 and the top, left shoulder, and right shoulder.

The support information output unit 14 may output face observation sheet information 15 recording the evaluation results for each divided area 102 instead of or in addition to the face observation sheet information 15 recording the evaluation results for each of the three areas, the top, left shoulder, and right shoulder. In addition, the format of the face observation sheet information 15, such as the observation items included in the face observation sheet information 15, the layout of the evaluation results, and the output format of the evaluation results (score format, table format, graph format, image overlay format, etc.), are not limited to the example in FIG. 20, and may be changed as appropriate, and may be changeable by a manager or worker, for example.

Figure 21 is a diagram showing a face sketch screen 160, which is an example of a screen display of face sketch information 16. The face sketch screen 160 displays the evaluation results of the face F for each observation item superimposed on a display screen that displays image data. In the example of Figure 21, the face sketch screen 160 includes a face information column 161 that displays information about the face F, an image display area 162 that displays image data of the face F, a display item selection column 163 that selects which of the face area 101, the division area 102, and various observation items to display in the image display area 162, and a display marker 164 that displays the evaluation results of the face F in the image display area 162 according to the selection item in the display item selection column 163.

The working face information column 161 displays, for example, the site name, site location, observation date and time, etc. The image display area 162 displays image data acquired by the image data acquisition unit 11. The image display area 162 is configured to allow the image to be enlarged or reduced, and is configured to allow sketch operations on the screen by workers or managers. The image display area 162 may display an image before color correction is performed by the color correction unit 110, or an image after color correction is performed.

The display item selection field 163 has options for observation items, such as strike and dip (crack dominance direction), rock group, compressive strength, weathering, crack spacing, crack shape, crack condition, continuous cracks, man-made objects, and areas at risk of skin falling.

The display marker 164 draws a frame around the outer periphery of the face area 101 and the division area 102, and displays the evaluation results of each observation item. In this case, the display marker 164 may display the evaluation results of each observation item by superimposing them at the corresponding position on the image, or may display them so that the corresponding relationship can be seen using a leader line or the like. In addition, the display form (color, pattern, etc.) of the display marker 164 is not limited to the example in FIG. 21 and may be changed as appropriate, and may be changeable by a manager or worker, for example.

As described above, according to the face evaluation support device 4 of this embodiment, the face extraction unit 130 extracts the face region 101 corresponding to the face F based on at least one of the image data and the point cloud data, the roughness quantification unit 131 quantifies the roughness of the face surface in the face region 101 as a roughness feature based on the point cloud data, the region division unit 132 divides the face region 101 into divided regions 102 based on the distribution of the roughness feature, and the divided region evaluation processing unit 133 evaluates the observation items related to the face F for each divided region 102. Therefore, the face region 101 corresponding to the face F is divided into divided regions 102, and the observation items related to the face F are evaluated for each divided region 102. Therefore, the evaluation quality can be improved when evaluating the face F.

In addition, by providing the section area evaluation processing unit 133 with the respective units 133A to 133E as described above, it is possible to evaluate each observation item for each section area 102. Furthermore, by providing the face area evaluation processing unit 134 with the respective units 134A to 134G as described above, it is possible to evaluate the entire face area 101 for each observation item.

Other Embodiments
The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention. All such modifications are included in the technical concept of the present invention.

In the above embodiment, the face evaluation support program 10 has been described as being stored in the storage unit 40. In contrast, the face evaluation support program 10 may be provided by being recorded in a computer-readable storage medium such as a CD-ROM or DVD as a file in an installable or executable format. The face evaluation support program 10 may also be provided by being downloaded from an external device via the network 7. The various machine learning models used in the face evaluation unit 13 may also be stored in the storage unit 40, or may be stored in an external device and accessed via the network 7.

In the above embodiment, a case where a convolutional neural network is adopted as a specific method of machine learning using a machine learning model has been described, but the machine learning model may adopt any other machine learning method (including not only supervised learning but also unsupervised learning and reinforcement learning). Examples of other machine learning methods include tree types such as decision trees and regression trees, ensemble learning such as bagging and boosting, other neural network types (including deep learning) such as recurrent neural networks, clustering types such as hierarchical clustering, non-hierarchical clustering, k-nearest neighbors, and k-means, multivariate analysis such as principal component analysis, factor analysis, and logistic regression, and support vector machines.

1 ... face evaluation support system, 2 ... image capturing device, 3 ... three-dimensional shape measuring device,
4... Face evaluation support device, 5... Worker terminal device, 6... Manager terminal device, 7... Network,
10... Face evaluation support program,
11: image data acquisition unit, 12: point cloud data acquisition unit, 13: face evaluation unit,
14... Support information output unit, 15... Face observation sheet information, 16... Face sketch information,
20...Color sample,
40: storage unit, 41: control unit, 42: input unit, 43: display unit, 44: communication unit,
45: external device I/F unit, 46: media input/output unit,
100: target area, 101: face area, 102: division area, 110: color correction unit,
130... face extraction unit, 131... roughness quantification unit, 132... area division unit,
133...partition area evaluation processing unit, 133A...crack dominant direction determination unit,
133B... rock group determination unit, 133C... fracture interval determination unit,
133D: Crack shape determination unit; 133E: Evaluation category inference unit;
134... face area evaluation processing unit, 134A... first continuity crack detection unit,
134B: second continuity crack detection unit; 134C: third continuity crack detection unit;
134D: artificial object detection unit; 134E: first skin fall risk prediction unit;
134F: second skin fall risk part prediction unit; 134G: skin fall risk part evaluation unit;
150... Face observation sheet, 151... Face information column, 152... Evaluation category column,
160 ... Face sketch screen, 161 ... Face information column, 162 ... Image display area,
163: Display item selection field, 164: Display marker, 400: Face management database

Claims (14)

an image data acquisition unit that acquires image data of a target area including a tunnel face;
a point cloud data acquisition unit that acquires point cloud data obtained by measuring a three-dimensional shape of the target area;
A face evaluation unit that evaluates the face based on the image data and the point cloud data;
A support information output unit that outputs support information based on the evaluation result by the face evaluation unit,
The face evaluation unit is
A face extraction unit that extracts a face area corresponding to the face from the target area based on at least one of the image data and the point cloud data;
a roughness quantification unit that quantifies the roughness of the face in the face region as a roughness feature amount along a plurality of scanning lines set in a plurality of scanning directions with respect to the face region based on the point cloud data;
a region division unit that calculates a representative value of the roughness feature amount for each intersection point where a plurality of the scanning lines intersect, and divides the face region into divided regions based on a distribution of the representative value for each intersection point;
A divided area evaluation processing unit is provided for evaluating the face for each divided area into which the face area is divided by the area division unit, and outputting the evaluation result as the evaluation result.
Face evaluation support device. The segmented area evaluation processing unit includes:
Each normal vector of a triangular network composed of the point cloud data corresponding to the face region is polar projected onto a stereo net for each of the divided regions;
A fracture dominant direction determination unit that determines a fracture dominant direction indicating a strike and inclination of a fracture dominant on the face for each of the divided regions based on a distribution of plots polar-projected onto the stereonet;
The face evaluation support device according to claim 1. The roughness quantifying unit is
As the roughness feature amount, an amplitude value and a wavelength of a roughness approximation model obtained by approximating the point cloud data corresponding to the face region by a periodic function along a plurality of the scanning lines are quantified;
The segmented area evaluation processing unit includes:
A rock group determination unit is provided for determining a rock group indicating a type of rock constituting the face for each divided area based on the amplitude value and the wavelength.
The face evaluation support device according to claim 1 or 2. The segmented area evaluation processing unit includes:
performing an edge detection process for detecting edges in the image data corresponding to the divided region;
a crack spacing determination unit that determines the crack spacing for each of the divided regions by referring to correspondence information that defines a relationship between the edge density and the crack spacing based on the edge density of the edges detected by the edge detection process;
The face evaluation support device according to any one of claims 1 to 3. The roughness quantifying unit is
As the roughness feature amount, a wavelength of a roughness approximation model obtained by approximating the point cloud data corresponding to the face region by a periodic function along a plurality of the scanning lines is quantified;
The segmented area evaluation processing unit includes:
a crack shape determining unit that determines a crack shape for each of the divided regions based on the wavelength characteristics according to the scanning direction of the scanning line when the roughness approximation model is approximated,
The face evaluation support device according to any one of claims 1 to 4. The segmented area evaluation processing unit includes:
The image data corresponding to the divided area is input to a machine learning model in which machine learning is performed using training data including learning image data of the face and a correct answer label to which an evaluation category related to the observation item is assigned for the face, thereby inferring the evaluation category for each divided area.
The face evaluation support device according to any one of claims 1 to 5. The face evaluation unit is
a first continuous crack detection unit that detects continuous cracks formed continuously on the face of the tunneling tunnel based on the point cloud data;
The first continuity crack detection unit includes:
extracting edge candidates constituting a part of the continuous crack from each side of a triangular net composed of the point cloud data corresponding to the face region based on an angle formed by normal vectors of two triangles including a common side, and detecting the continuous crack present in the face region by determining the continuity between the candidates;
The face evaluation support device according to any one of claims 1 to 6. The face evaluation unit is
a second continuous crack detection unit that detects continuous cracks formed continuously on the face of the tunneling tunnel based on the image data;
The second continuity crack detection unit includes:
The image data corresponding to the face region is input to a machine learning model in which machine learning is performed using training data including learning image data of the face and correct labels assigned to the continuous cracks formed in the face, thereby detecting the continuous cracks present in the face region.
The face evaluation support device according to any one of claims 1 to 7. The segmented area evaluation processing unit includes:
Each normal vector of a triangular network composed of the point cloud data corresponding to the face region is polar projected onto a stereo net for each of the divided regions;
A fracture dominant direction determination unit that determines a fracture dominant direction indicating a strike and inclination of a fracture dominant on the face for each of the divided regions based on a distribution of plots polar-projected onto the stereonet,
The face evaluation unit is
a third continuous crack detection unit that detects continuous cracks formed continuously on the face of the tunneling tunnel based on the image data and the point cloud data;
The third continuity crack detection unit includes:
performing an edge detection process for detecting edges on the image data corresponding to the face region;
The edge detected by the edge detection process is fitted to the crack dominant direction for each of the divided areas determined by the crack dominant direction determination unit, thereby detecting the continuous crack present in the face area.
The face evaluation support device according to any one of claims 1 to 8. The face evaluation unit is
an artifact detection unit that detects an artifact present in the target area based on the image data;
The artifact detection unit includes:
The image data is divided into the image data corresponding to the face region and the image data corresponding to other than the face region, and each of the image data is input to a machine learning model in which machine learning is performed using training data including learning image data of the face and correct answer labels assigned to artificial objects installed at or around the face, thereby detecting the artificial objects present in the face region and the artificial objects present outside the face region.
The face evaluation support device according to any one of claims 1 to 9. The segmented area evaluation processing unit includes:
Each normal vector of a triangular network composed of the point cloud data corresponding to the face region is polar projected onto a stereo net for each of the divided regions;
A fracture dominant direction determination unit that determines a fracture dominant direction indicating a strike and inclination of a fracture dominant on the face for each of the divided regions based on a distribution of plots polar-projected onto the stereonet;
The face evaluation unit is
A first skin flaking risk prediction unit predicts a skin flaking risk location based on the point cloud data,
The first skin loss risk part prediction unit,
The crack prominent direction for each of the divided areas determined by the crack prominent direction determination unit is projected onto a stereo net as a great circle;
When a closed polygon is formed by the predominant crack directions extending in at least three directions, the range of the face corresponding to the closed polygon is predicted as the skin fall danger area .
The face evaluation support device according to any one of claims 1 to 10. The face evaluation unit is
A second skin flaking risk part prediction unit that predicts the skin flaking risk part based on the image data;
a skin loss danger part evaluation part that predicts the skin loss danger part based on a prediction result by the first skin loss danger part prediction part and a prediction result by the second skin loss danger part prediction part;
The second skin loss risk prediction unit is
The image data corresponding to the face region is input to a machine learning model in which machine learning is performed using training data including learning image data of the face and a correct answer label assigned to the face-fall danger area at the face, thereby predicting the face-fall danger area ;
The skin fall risk area evaluation unit is
the skin loss risk area predicted by the first skin loss risk area prediction unit and the skin loss risk area predicted by the second skin loss risk area prediction unit are compared, and a skin loss risk area having a high degree of agreement as a result of the comparison is determined to be at high risk ;
The face evaluation support device according to claim 11. The image data acquisition unit
A color correction unit performs color correction on color image data obtained by photographing the target area in color, the image data including the face and a color sample having a predetermined black and white pattern,
The color correction unit includes:
converting the color image data into a grayscale image, and arranging all pixels constituting the grayscale image in order of brightness to obtain the brightness of a pixel located in the middle as a brightness median value;
Classifying all the pixels into a high-brightness pixel group consisting of pixels having a brightness higher than the intermediate brightness value and a low-brightness pixel group consisting of pixels having a brightness equal to or lower than the intermediate brightness value;
calculating average values of the three RGB components for a pixel group corresponding to a white portion of the color sample, selecting a maximum value among the average values, and adding difference values between the maximum value and each of the average values as correction values for the high-lightness pixel group to each of the pixels of the high-lightness pixel group for each of the three components, thereby color-correcting the high-lightness pixel group;
calculating average values of the three components for a pixel group corresponding to a black portion of the color sample, selecting a minimum value from among the average values, and subtracting each difference value between the minimum value and each average value from each pixel of the low-lightness pixel group for each of the three components as a correction value for the low-lightness pixel group, thereby color-correcting the low-lightness pixel group;
The face evaluation support device according to any one of claims 1 to 12. An image data acquisition step of acquiring image data of a target area including a tunnel face;
a point cloud data acquisition step of acquiring point cloud data obtained by measuring a three-dimensional shape of the target area;
A face evaluation process for evaluating the face based on the image data and the point cloud data;
A support information output process for outputting support information based on the evaluation result by the face evaluation process,
The face evaluation process includes:
A face extraction process of extracting a face area corresponding to the face from the target area based on at least one of the image data and the point cloud data;
a roughness quantification step of quantifying the roughness of the face in the face region as a roughness feature value along a plurality of scanning lines set in a plurality of scanning directions with respect to the face region based on the point cloud data;
A region dividing step of calculating a representative value of the roughness feature amount for each intersection point where a plurality of the scanning lines intersect, and dividing the face region into divided regions based on a distribution of the representative value for each intersection point;
For each divided area into which the face area is divided by the area division step, the face is evaluated for the observation items related to the face, and the evaluation result is output as the evaluation result.
A method to assist in face evaluation.

Images