JP7721086B2 - Detection device and detection method - Google Patents

Description

The present invention relates to a detection device and detection method for detecting the excavation position of an excavator.

Patent Document 1 describes a method for evaluating the natural ground ahead of the tunnel face using drilling energy measured by a drill jumbo during the excavation of a mountain tunnel. Patent Document 1 also describes how a computer-controlled drill jumbo can automatically obtain drilling data such as drilling position, drilling energy, and drilling direction angle data during drilling.

Japanese Patent Application Laid-Open No. 2021-127655

In the method described in Patent Document 1, various sensors are installed in the drill jumbo to sense the operation of the drill jumbo, thereby automatically obtaining the drilling position.

However, obtaining the drilling position from a sensor attached to the drill jumbo requires multiple sensors. It is also necessary to measure the sensor mounting position and the shape of the drill jumbo and calculate the position of the drill jumbo's tip. As such, using sensors to sense the operation of the drill jumbo and obtain the drilling position is costly, and there is a need for a cheaper method of obtaining the drilling position.

The present invention aims to obtain the excavation position of an excavator in an inexpensive manner.

The present invention is a detection device that detects the excavation position of an arm-shaped excavator in the ground, and includes an image acquisition unit that acquires a target image of the ground and the excavator, a processing and detection unit that performs image processing on the target image to detect the tip of the excavator in the target image, an energy information acquisition unit that acquires information on the excavation energy value used to excavate the ground by the excavator in association with the target image, and a position identification unit that, when an excavation energy value equal to or greater than a predetermined threshold is acquired, identifies the position of the tip of the excavator detected in the associated target image as the excavation position.

The present invention is a detection method for detecting the excavation position of an arm-shaped excavator in the ground. It involves acquiring a target image of the ground and the excavator, applying image processing to the target image to detect the tip of the excavator within the target image, acquiring information on the excavation energy value used to excavate the ground by the excavator in association with the target image, and, when an excavation energy value equal to or greater than a predetermined threshold is acquired, identifying the position of the tip of the excavator detected within the associated target image as the excavation position.

This invention allows for inexpensive acquisition of excavation locations.

1 is a schematic diagram of a detection system according to an embodiment of the present invention; 1 is a block diagram showing a configuration of a detection system according to an embodiment of the present invention. 3 is a flowchart illustrating a detection method according to an embodiment of the present invention. 1 is a flowchart illustrating an evaluation method according to an embodiment of the present invention. 10A and 10B are diagrams illustrating a state in which an excavator is detected in a target image in a detection method according to an embodiment of the present invention. 10A and 10B are diagrams illustrating a method for estimating the position of the tip of an excavator in an object image in a detection method according to an embodiment of the present invention; FIG. 10 is a diagram showing a state in which a working face is divided into a plurality of regions in a target image in an evaluation method according to an embodiment of the present invention. 1A and 1B are diagrams illustrating evaluation images in an evaluation method according to an embodiment of the present invention.

The following describes a detection system 101 equipped with a detection device 100 and a detection method according to an embodiment of the present invention, with reference to the drawings.

The detection system 101 is applied to a work machine 102 that has an arm-shaped excavator 10 and excavates the ground using the excavator 10, and detects the excavation position of the ground using the excavator 10.

First, the overall configuration of the work machine 102 will be explained with reference to Figure 1.

In this embodiment, the work machine 102 is a drill jumbo that drills blast holes H for loading explosives into the face S of a tunnel T, which is the ground to be excavated. The work machine 102 includes an arm-shaped excavator 10 and a mobile carriage 1 on which the excavator 10 is mounted.

As shown in Figure 1, the excavator 10 has a rod 12 with a bit 11 attached to one end, a drifter 13 that supports the other end of the rod 12, and a guide cell 14 that guides the movement of the drifter 13. When the drifter 13 moves back and forth along the guide cell 14 by a drive mechanism (not shown), the rod 12 moves back and forth along the guide cell 14.

The drifter 13 is capable of applying rotational force and impact force to the rod 12. The bit 11 is pressed against the rock at the working face S, and the bit 11 is rotated and used to impact the rock, thereby drilling a blast hole H. The work machine 102 typically includes multiple excavators 10, but in this embodiment, for ease of explanation, only a single excavator 10 is shown.

The mobile carriage 1 supports the base end of an extendable boom 15 so that it can swing freely up and down and left and right, and the guide cell 14 of the excavator 10 is supported at the tip of the boom 15 so that it can swing freely up and down and left and right. The boom 15 swings and retracts relative to the mobile carriage 1 by a drive mechanism (not shown). The excavator 10 moves up and down and left and right as the boom 15 swings and retracts. The guide cell 14 of the excavator 10 swings relative to the boom 15 by a drive mechanism (not shown). The orientation of the excavator 10 (the orientation of the rod 12) changes as the guide cell 14 swings.

The excavator 10 is operated by an operator to drill the working face S and perforate the blast hole H. Normally, the mobile carriage 1 does not move while the excavator 10 is drilling the working face S, and the positional relationship between the mobile carriage 1 and the working face S does not change.

As shown in Figures 1 and 2, the detection system 101 includes a camera 20 as an imaging unit that images the working face S, a measurement unit 25 for acquiring the excavation energy value of the excavator 10, and a detection device 100 that detects the position of the hole drilled into the working face S by the excavator 10.

The camera 20 is attached to the mobile carriage 1 so as to be able to capture the entire face S being drilled by the excavator 10. The camera 20 continuously captures images of the face S to obtain image data in the form of a moving image. Note that the camera 20 may also be capable of intermittently capturing still images. The image data obtained by the camera 20 is input into the detection device 100.

The measurement unit 25 measures parameters representing the operating state of the excavator 10, which are necessary to obtain the excavation energy value of the excavator 10. The measurement unit 25 measures parameters representing the operating state of each of the multiple excavators 10 in a manner that allows them to be distinguished from one another. In this embodiment, the measurement unit 25 is composed of various sensors that measure the rotational force and impact force applied to the rod 12 by the drifter 13. For example, although not shown, the measurement unit 25 includes a torque sensor that detects the rotational force of the rod 12, a pressure sensor that detects the impact force of the rod 12, and a speed sensor that detects the movement speed of the rod 12. The measurement unit 25 measures the rotational force, impact force, and movement speed of the rod 12 in accordance with the interval (sampling interval) between still image captures by the camera 20. In other words, the measurement unit 25 measures each measurement item for each frame captured by the camera 20. The measurement results of the measurement unit 25 are input to the detection device 100. Note that the measurement unit 25 only needs to set the sampling interval so that it can obtain at least the excavation energy when the target image, described below, is captured.

The detection device 100 is composed of a computer equipped with a CPU (Central Processing Unit) that executes control programs, etc., a ROM (Read-Only Memory) that stores the control programs executed by the CPU, a RAM (Random Access Memory) that stores the results of CPU calculations, etc., and a communication device. The detection device 100 performs the various functions of the detection device 100 described in this specification by the CPU executing the control programs stored in the ROM. The detection device 100 may be composed of a single computer, or may be composed of multiple microcomputers and configured to distribute the various controls among the multiple computers.

The detection device 100 is mounted, for example, on the mobile dolly 1 and is connected to the camera 20 and measurement unit 25 via wired or wireless communication. Note that the detection device 100 is not limited to being mounted on the mobile dolly 1, and may be provided in a location other than the mobile dolly 1. For example, the detection device 100 may be configured as a server computer wirelessly connected to the camera 20 and measurement unit 25 via a network.

As shown in FIG. 2, the detection device 100 comprises an image acquisition unit 30 that acquires a target image from image data input from the camera 20; a processing and detection unit 31 that performs image processing on the target image to detect the tip of the excavator 10 in the target image; an energy information acquisition unit 32 that acquires information on the excavation energy value used to drill the working face S by the excavator 10 in association with the target image; a position identification unit 33 that identifies the excavation position by the excavator 10 in the target image based on the excavation energy value; and an energy calculation unit 34 that divides the imaged working face S into multiple regions and calculates the excavation energy value for each region based on the excavation energy value at one or more excavation positions included in each region. Note that the components of the detection device 100 shown in FIG. 2 are virtual units that represent the functions of the detection device 100 and do not represent physical existences.

The image acquisition unit 30 extracts still images at predetermined time intervals from image data input in video format. The extracted still images (hereinafter referred to as "target images") are subjected to image processing by the processing detection unit 31. The interval at which still images are extracted from the image data may be the same as the sampling interval at which the camera 20 captures still images (i.e., all frames of the video are extracted as target images), or may be longer than that sampling interval. In the detection method described below, each frame of the video is extracted as the target image.

The processing and detection unit 31 detects the front end, rear end, and entire excavator 10 in the target image using a trained model that has been machine-learned using a group of images in which the front end of the excavator 10 is identified as training data. The front end of the excavator 10 here refers to the front end of the rod 12 to which the bit 11 is attached.

The trained model is constructed by having an operator label the front end, rear end, and entirety of the excavator 10 in a group of images taken in advance of the working face S and the excavator 10, and then having a computer perform machine learning using deep learning using the labeled group of images as training data. The trained model is also trained to be able to identify multiple excavators 10 individually. For example, by attaching markers to each excavator 10 and using a group of images labeled based on the markers as training data, a trained model that can identify excavators 10 can be generated.

Image processing and object recognition by the processing and detection unit 31 extracts rectangular regions (bounding boxes) B1, B2, and B3 corresponding to the front end, rear end, and entire excavator 10, respectively (see Figure 5). Furthermore, if the processing and detection unit 31 is unable to extract the front end of the excavator 10, it has the function of estimating and detecting the position of the front end based on the extracted rear end and entire excavator. Details of this function will be explained later.

The energy information acquisition unit 32 acquires the excavation energy value of the excavator 10 at the time the target image was captured based on the measurement results of the measurement unit 25. The excavation energy value is calculated based on the rotational force, impact force, and movement speed of the excavator 10. A well-known method can be used to calculate the excavation energy value from the rotational force, impact force, and movement speed of the excavator 10, so a detailed explanation will be omitted. The target image and the excavation energy value at the time the target image was captured are stored in association with each other and input to the position identification unit 33.

If the drilling energy value of the target image input from the energy information acquisition unit 32 is greater than a preset threshold, the position identification unit 33 identifies the position of the tip of the excavator 10 in the target image as the drilling position relative to the working face S. In this way, the excavation position in the target image is identified. The threshold is set to a value greater than zero so as to avoid any influence from fluctuations in the excavation energy value caused by sensor errors or noise in the measurement unit 25 when the excavator 10 is not excavating the working face S.

Once the position identification unit 33 identifies the excavation position within the target image, it compares the target image with a reference image in which the shape of the drill face S has been identified in advance, and identifies the shape of the drill face S within the target image. This identifies the relative positional relationship between the detected excavation position and the drill face S.

The energy calculation unit 34 creates a grid on the target image and divides the working face S in the target image into multiple regions. The energy calculation unit 34 calculates the excavation energy value for each divided region based on the position of the working face S and the excavation position in the target image identified by the position identification unit 33, and the excavation energy value for excavating the excavation position. The excavation energy value for each region is a single excavation energy value (representative value) corresponding to each region. If multiple blast holes H are created within a region, the excavation energy value for each region can be the average of the excavation energy values for excavating each blast hole H, or the maximum value of the excavation energy values of the blast holes H in the region. The evaluation method for the working face S described below will be described using the maximum value as the representative value.

The energy calculation unit 34 also generates an evaluation image by assigning color and brightness to multiple divided regions within the target image according to the magnitude of the excavation energy value. The generated evaluation image is output from the detection device 100 to an external monitor. This allows the worker to understand the hardness or softness of the working face S by checking the distribution of excavation energy values expressed by differences in color and brightness in the evaluation image.

Next, with reference to Figures 3 to 8, we will explain how the detection system 101 detects the excavation position and evaluates the working face S.

Figure 3 is a flowchart showing the excavation position detection method executed by the detection device 100, and Figure 4 is a flowchart showing the evaluation method for the face S. The detection device 100 detects the excavation position of the excavator 10 by performing image processing on the target image, and evaluates the condition of the face S based on the excavation position and the excavation energy value required for excavation.

First, the method for detecting the excavation position will be described. The detection device 100 performs the excavation position detection process shown in Figure 3 at a predetermined processing interval. In this embodiment, the processing interval of the detection process performed by the detection device 100 matches the time interval between frames captured by the camera 20. In other words, the detection system 101 performs the process shown in Figure 3 for each frame of the video captured by the camera 20. Note that the processing interval performed by the detection system 101 only needs to be set to be equal to or longer than the time interval between frames.

In step S10, one frame of still image is extracted from the video captured by the camera 20 and acquired as the target image.

In step S11, the trained machine learning model is used to extract the front end, rear end, and entire area of the excavator 10 from the target image. Specifically, as shown in Figure 5, a rectangular area B1 representing the front end of the excavator 10, a rectangular area B2 representing the rear end, and a rectangular area B3 representing the entire excavator 10 are extracted.

Here, because the tip of the excavator 10 comes into contact with the working face S and excavates, it is difficult to extract an area within the target image using a machine learning model to recognize objects within an image compared to the rear end or the entire excavator. For this reason, while the rear end and the entire excavator 10 may be detected from the target image, the tip may not be detected. However, even if the tip is not detected, the entire excavator 10 is easier to detect than the tip, as it exists within a certain range within the target image and its features can be extracted.

In such cases, the detection system 101 detects the tip of the excavator 10 by estimating the position of the tip based on the relative positional relationship between the rear end of the excavator 10 and the entire excavator. Specifically, as shown in FIG. 6, rectangular area B2 representing the rear end of the excavator 10 is typically located within rectangular area B3 representing the entire excavator 10. The tip of the excavator 10 is located at the end of the body extending from the rear end of the excavator 10. Therefore, if only the tip of the excavator 10 is not detected, the detection device 100 estimates the position P1 of the tip in the target image to be point-symmetrical to the center point P3 of rectangular area B3 representing the entire excavator 10 and the center point P2 of rectangular area B2 at the rear end. In other words, on the imaginary line (the two-dot chain line in Figure 6) connecting the center point P3 of rectangular area B3 representing the entire excavator 10 and the center point P2 of rectangular area B2 representing the rear end, point P1 that is equidistant from the center point P2 of the entire excavator 10 as well as the center point P3 of the rear end is detected as the tip. This makes it possible to detect the hole-boring position of the excavator 10 with a certain degree of accuracy, even if the tip of the excavator 10 cannot be directly detected from the target image.

Steps S12 and S13 are executed in parallel with the execution of steps S10 and S11. Note that steps S10 and S11 and steps S12 and S13 do not have to be processed in parallel, but may be processed serially, and the order of these steps may be either before or after. In step S12, the measurement results (measurement results corresponding to the target image) obtained by the measurement unit 20 when the target image was captured are obtained.

In step S13, the excavation energy value of the excavator 10 is calculated from the measurement results obtained in step S12.

After the excavator 10 is extracted in step S11 and the excavation energy value is calculated in step S13, it is determined in step S14 whether the excavation energy is equal to or greater than a predetermined threshold. If the excavation energy value is less than the threshold, the process ends. If the excavation energy is equal to or greater than the threshold, the process proceeds to step S15, where the target image (more specifically, the position of the tip of the excavator 10 within the target image) and the excavation energy value are associated and stored. The position of the tip of the excavator 10 at this time is detected as the hole-boring position by the excavator 10. When step S15 is completed, the process ends.

The detection device 100 continues to detect drilling positions in this manner, for example, until a predetermined number of blast holes H have been formed. Once the predetermined number of blast holes H have been detected, the detection device 100 performs the evaluation process shown in Figure 4 to evaluate the working face S. Note that the detection device 100 may also be configured to perform the process shown in Figure 4 when an operator inputs an operation, for example. In this way, the timing for performing the process shown in Figure 4 can be set as desired.

As shown in FIG. 4, in step S20, the shape of the ground (shape of the face S) to be excavated in the target image is identified. Specifically, a reference image in which the shape of the face S (position within the image) is identified is stored in advance in the detection system 101. The reference image is, for example, an image showing the design shape of the face S for the excavation of tunnel T. The shape of the face S in the target image is identified by fitting the target image to the reference image. Fitting involves moving and scaling the reference image so that the face S in the reference image overlaps the face S in the target image, thereby superimposing the target image and the reference image. Fitting the reference image to the target image may be performed by the detection device 100 or manually by an operator. As a result, the excavation position, which was a coordinate position in the target image, is recognized as a position relative to the face S.

Next, in step S21, as shown in Figure 7, the working face S in the target image is divided into multiple regions using grid lines. Note that the spacing and shape of the grid lines are not limited to the example shown in Figure 6 and can be set arbitrarily. Furthermore, each point shown in Figure 7 indicates the excavation position of the excavator 10.

In step S22, one excavation energy value (representative value) is calculated for each cell C divided by grid lines. If multiple blast holes H have been formed within cell C, the representative excavation energy value is the maximum value of the excavation energy values for the multiple blast holes H. Furthermore, if no blast holes H have been formed within cell C, the excavation energy value for that cell C is calculated as 0 (zero).

Next, in step S23, an evaluation image is generated in which cells C are color-coded according to the magnitude of the excavation energy value (see Figure 8). Then, in step S24, the evaluation image is output, for example, to an external monitor, and the process ends. This allows the worker to determine the hardness or softness of the ground by checking the evaluation image.

The above embodiment provides the following advantages:

In this embodiment, the tip of the excavator 10 is detected by image processing, and it is possible to determine whether the excavator 10 is excavating the ground based on the excavation energy value. In other words, the tip of the excavator 10 can be detected by image processing while it is excavating the ground, making it possible to identify the excavation position relative to the ground. In this way, because the position of the tip of the excavator 10 can be identified by image processing without using a sensor, it is possible to reduce the number of sensors that sense the operation of the excavator 10, and it is possible to obtain the excavation position more inexpensively.

In addition, in this embodiment, the position of the tip of the excavator 10 when the excavation energy value is calculated is detected as the excavation position only if the excavation energy value is equal to or greater than a threshold value. An evaluation image is generated based on the excavation position and the excavation energy value. This makes it possible to grasp the hardness of the face S. By grasping the hardness of the face S, it is possible to manage the charge for the blast hole H, predict the hardness of the ground to be excavated (such as the new face S that will be created after blasting), evaluate and manage the support work to be installed in the tunnel T, and so on.

Next, we will explain a modified example of this embodiment.

In the above embodiment, the front and rear ends of the excavator 10 are detected in the target image using a machine-learned learning model. The detection system 101 may calculate the angle of the excavator 10, and therefore the drilling angle, based on the front and rear ends. Because the overall length of the excavator 10 can be known in advance, it is possible to calculate the drilling angle from the overall length of the excavator 10 and the positions of the front and rear ends of the excavator 10. The drilling angle calculated in this manner may be used, for example, to evaluate the accuracy of drilling.

In addition, in the above embodiment, the detection device 100 is applied to a drill jumbo that drills the working face S, but this is not limited to this. For example, it may also be applied to a road header that similarly excavates the ground using an excavator 10.

Furthermore, in the above embodiment, when the digging energy value reaches or exceeds the threshold value, the position of the tip of the excavator 10 at that time is acquired as the digging position. This configuration does not prevent the position of the tip of the excavator 10 from being stored when the digging energy value does not reach the threshold value.

In addition, in the above embodiment, the shape of the tunnel face S in the target image is identified by comparing (fitting) the target image with the reference image, and the relative position of the blast hole H with respect to the tunnel face S is determined. In the above embodiment, the reference image was based on the design shape of the tunnel face S. However, for example, the tunnel face S may actually be imaged, and the shape of the tunnel face S in the image may be detected by image processing or manually by an operator to serve as the reference image. Furthermore, determining the relative position of the blast hole H with respect to the tunnel face S is not limited to comparing the target image with the reference image. Even without using the reference image, the shape of the tunnel face S in the target image captured by the camera 20 can be identified by previously determining the actual shape of the tunnel face S and the relative position of the camera 20 with respect to the tunnel face S. Furthermore, image processing can be performed on the target image to detect the boundary between the tunnel face S and the inner wall of the tunnel T, thereby determining the shape of the tunnel face S in the target image.

In addition, in the above embodiment, the processing and detection unit 31 detects the excavator 10 in the target image using a machine-learned learning model, but detection of the excavator 10 is not limited to image processing using a learning model. When using a trained model, if the ground to be excavated is updated or changed, it is desirable to label the image of the ground to create new training data and then additionally train the new training data.

The above describes embodiments of the present invention, but these embodiments merely illustrate some of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments.

The series of processes performed by the detection device 100 described above may be provided as a program for execution by a computer. Furthermore, the program for executing the series of processes described above may be provided on a storage medium readable by the detection device 100. Alternatively, the program may be provided to the detection device 100 via a network line. Furthermore, the various programs executed by the detection device 100 may be stored on a non-transitory storage medium, such as a CD-ROM.

100 Detection device 10 Excavator 20 Camera (imaging unit)
25 Measurement unit 30 Image acquisition unit 31 Processing detection unit 32 Energy information acquisition unit 33 Position identification unit 34 Energy calculation unit

Claims (6)

A detection device for detecting an excavation position of a ground by an arm-shaped excavator,
an image acquisition unit that acquires a target image obtained by capturing an image of the ground and the excavator;
a processing and detection unit that applies image processing to the target image to detect the tip of the excavator in the target image;
an energy information acquisition unit that acquires information on an excavation energy value used to excavate the ground by the excavator in association with the target image;
and a position specifying unit that specifies, when the excavation energy value is equal to or greater than a predetermined threshold, the position of the tip of the excavator detected in the associated target image as the excavation position.
A detection device characterized by: 2. The detection device according to claim 1,
the processing and detection unit detects the tip of the excavator using a trained model that has been machine-learned using training data.
A detection device characterized by: 3. The detection device according to claim 1 or 2,
the processing and detection unit is configured to detect the front end, rear end, and entirety of the excavator, and if the front end cannot be detected, detects the position of the front end from the positional relationship between the rear end and the entirety;
A detection device characterized by: 3. The detection device according to claim 1 or 2,
the position specifying unit specifies the shape of the ground in the target image by comparing the target image with a reference image in which the shape of the ground in the image is specified, and specifies a positional relationship between the ground and the excavation position;
A detection device characterized by: 5. The detection device according to claim 4,
The ground is divided into a plurality of regions, and an energy calculation unit is further provided which calculates the excavation energy value in each region based on the excavation energy value at one or more excavation positions included in each region.
A detection device characterized by: A detection method for detecting an excavation position of a ground by an arm-shaped excavator, comprising:
acquiring a target image of the ground and the excavator;
applying image processing to the target image to detect the tip of the excavator in the target image; and acquiring information on an excavation energy value for excavating the ground by the excavator in association with the target image.
When the excavation energy value is equal to or greater than a predetermined threshold, the position of the tip of the excavator detected in the associated target image is identified as the excavation position.
A detection method characterized by: