JP7465664B2 - Work machine monitoring device and work machine monitoring method - Google Patents

Description

The present invention relates to a monitoring device for a work machine and a monitoring method for a work machine.

Patent Document 1 describes an example of a monitoring system for avoiding contact between construction machinery and people. The monitoring system described in Patent Document 1 captures images of the surroundings of the construction machinery with a stereo camera attached to the construction machinery, calculates the distance to the subject based on the parallax of the stereo camera, and recognizes people and helmets contained in the captured images using image recognition processing to calculate the distance to the people and helmets. This monitoring system then executes processing to avoid contact, such as issuing a warning or stopping the operation of the construction machinery, depending on the results of a distance-based determination process. In this monitoring system, image recognition processing, distance calculation processing, and distance-based determination processing are performed for each captured image.

JP 2019-157497 A

As described in Patent Document 1, when the recognition of people, etc., the calculation of distance, and the determination process based on the distance are performed for each acquired image, there is a problem that, for example, when there is a large error in the calculation of the distance, the determination result is inverted for each image, making it impossible to perform the determination process stably.

The present invention has been made in consideration of the above circumstances, and aims to provide a work machine monitoring device and a work machine monitoring method that can stably perform distance-based determination processing.

In order to solve the above problem, one aspect of the present invention is a monitoring device for a work machine that includes a distance acquisition unit that recognizes an object present within a target range and acquires a distance acquisition value that indicates the distance to the recognized object, a distance calculation unit that calculates a distance calculation value that indicates the latest distance to the object based on a plurality of the distance acquisition values in a time series, and an output unit that compares the distance calculation value with a predetermined threshold value and outputs a predetermined signal based on the comparison result.

According to one aspect of the present invention, a distance calculation value that represents the latest distance to an object is calculated based on multiple distance acquisition values in a time series, so that distance-based determination processing can be performed stably.

1 is a block diagram showing a schematic configuration example of a monitoring system according to an embodiment of the present invention; 2 is a plan view showing a schematic example of an installation of the imaging device 2 shown in FIG. 1. 1. FIG. 4 is a plan view showing a schematic diagram of another installation example of the imaging device 2 shown in FIG. 4 is a flowchart showing an example of the operation of the monitoring device 3 shown in FIG. 1 . 5 is a flowchart showing an example of the process of step S7 shown in FIG. 4. 3A to 3C are diagrams showing images for explaining an example of the operation of the monitoring device 3 shown in FIG. 1 . 2 is a schematic diagram for explaining an example of the operation of the monitoring device 3 shown in FIG. 1 . 2 is a schematic diagram for explaining an example of the operation of the monitoring device 3 shown in FIG. 1 . 2 is a schematic diagram for explaining an example of the operation of the monitoring device 3 shown in FIG. 1 . 2 is a schematic diagram for explaining an example of the operation of the monitoring device 3 shown in FIG. 1 . 14 is a flowchart showing another example of the process of step S7 shown in FIG. 4 in conjunction with FIG. 12 and FIG. 13. 14 is a flowchart showing another example of the process of step S7 shown in FIG. 4 in conjunction with FIG. 11 and FIG. 13. 13 is a flowchart showing another example of the process of step S7 shown in FIG. 4 in conjunction with FIG. 11 and FIG. 12. 14A to 14C are schematic diagrams for explaining another example of the process of step S7 shown in FIG. 4, which are shown in FIGS. 11 to 13.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the same or corresponding components in each drawing will be designated by the same reference numerals and their explanations will be omitted as appropriate.

Figure 1 is a block diagram showing an example of the schematic configuration of a monitoring system 1 according to an embodiment of the present invention. Figures 2 and 3 are plan views showing a schematic example of an installation of the imaging device 2 shown in Figure 1. Figure 4 is a flowchart showing an example of the operation of the monitoring device 3 shown in Figure 1. Figure 5 is a flowchart showing an example of the processing of step S7 shown in Figure 4. Figure 6 is a diagram showing example images for explaining an example of the operation of the monitoring device 3 shown in Figure 1. Figures 7 to 10 are schematic diagrams for explaining an example of the operation of the monitoring device 3 shown in Figure 1.

The monitoring system 1 shown in FIG. 1 includes an imaging device 2, a monitoring device 3, and an output device 4. The monitoring system 1 shown in FIG. 1 is configured as a system in which the imaging device 2 attached to the hydraulic excavator 101 shown in FIG. 2 captures an image of the imaging range OA (target range) of the imaging device 2, the monitoring device 3 recognizes a person (target object) present in the imaging range OA based on the captured image, and estimates the distance to the person. If the estimated distance falls below a predetermined threshold, the monitoring device 3 outputs a signal for an alarm or the like to the output device 4.

Note that the monitoring device 3 of this embodiment does not limit the object to be monitored to a person. For example, in other embodiments, the object can be another work machine, a passenger vehicle, etc. Furthermore, the device that acquires information to recognize the object in the target range and the distance to the object is not limited to an imaging device such as a monocular camera or a stereo camera, but can be a device that measures the shape, direction, distance, etc. of the object using ultrasound, infrared rays, microwaves, lasers, etc., or a combination thereof. However, in this embodiment, as an example, the object is a person, and the device that acquires information to recognize the object in the target range and the distance to the object is an imaging device having a monocular camera. In this case, the target range is the imaging range OA of the imaging device 2, and when the imaging device 2 captures the imaging range OA, the captured image corresponds to the target range.

In FIG. 2, the hydraulic excavator 101 includes a vehicle body 101B consisting of an upper rotating body 103 and a traveling device 105, and a work machine 102 having a bucket 108 at its tip. The hydraulic excavator 101 travels by driving the traveling device 105. The work machine 102 is attached to the upper rotating body 103 so that it can swing. The upper rotating body 103 rotates around a rotation center RC. A driver's cab 104 is also installed on the upper rotating body 103.

The imaging device 2 is attached to the upper rotating body 103, captures moving images of subjects present in the imaging range OA, and outputs a signal representing the captured moving image to the monitoring device 3. Note that there may be multiple imaging devices 2, for example, as shown in FIG. 3. In the example shown in FIG. 3, four imaging devices 2, 2-1, 2-2, and 2-3 are attached to the upper rotating body 103 with their imaging directions differing by 90 degrees each. In the example shown in FIG. 3, a 360-degree range can be monitored by capturing images of the imaging ranges OA, OA1, OA2, and OA3 using the imaging devices 2, 2-1, 2-2, and 2-3.

On the other hand, the monitoring device 3 shown in FIG. 1 can be configured using hardware such as a computer, such as a microcomputer or a programmable device having a CPU (central processing unit), and peripheral devices of the computer. The monitoring device 3 of this embodiment has a distance acquisition unit 5, a distance calculation unit 6, and an output unit 7 as a functional configuration configured by a combination of hardware such as a computer and software such as a program executed by the computer.

The distance acquisition unit 5 has an input unit 51, an object recognition unit 52, a tracking unit 53, a recognition result filter 54, a distance estimation unit 55, a coordinate conversion unit 56, and a memory unit 57. The memory unit 57 stores an image file 571, a recognition model 572, and a distance estimation model 573. The distance acquisition unit 5 recognizes a person (object) present in the imaging range OA (object range) of the imaging device 2, and acquires a distance acquisition value that indicates the distance to the recognized person.

The input unit 51 inputs an image signal representing an image captured by the imaging device 2 of the imaging range OA (target range), and stores the image in the storage unit 57 as an image file 571 including, for example, multiple still images for each frame. In this example, the monitoring device 3 executes the recognition process and determination process described below for each frame of the moving image captured by the imaging device 2. However, the monitoring device 3 may execute the recognition process and determination process, for example, for each multiple frames.

The object recognition unit 52 recognizes a person (object) from the image input by the input unit 51 using the recognition model 572, and outputs coordinate information representing the position of the area including the recognized person. The recognition model 572 is, for example, a trained model with a neural network as an element, and the weighting coefficients between the neurons of each layer of the neural network are optimized by machine learning so that a desired solution is output for a large amount of input data. The recognition model 572 is, for example, composed of a combination of a program that performs calculations from input to output and weighting coefficients (parameters) used in the calculations. The recognition model 572 is trained by machine learning using a data set that corresponds information representing multiple images and information representing the coordinates of areas including a person when each image includes a person, as teacher data, so that an image is input, and if the image includes a person, the coordinate information of the area including the person is output as a solution. FIG. 6 shows an image 21 as an example of an image input to the object recognition unit 52. When the image 21 shown in FIG. 6 is input, the object recognition unit 52 calculates an area 91 (hereinafter referred to as the detection area 91) that includes a person 81 included in the image 21, as shown in FIG. 7, for example, and calculates and outputs the coordinate information (X1, Y1) and (X2, Y2) of each pixel in the upper left and lower right of the detection area 91.

When the object recognition unit 52 detects a person in the previous frame but mistakenly fails to detect the person in the next frame, the tracking unit 53 estimates the position of the detection area that should be detected in the frame where the person was not detected (this is called tracking processing). In this embodiment, when the object recognition unit 52 does not recognize a person (object), the tracking unit 53 sets the coordinate information of the area including a partial image similar to the image in the detection area including the person in the previous image (referred to as the area image) in the image in which the person is not recognized as the coordinate information of the detection area including the person in the image in which the person is not recognized. For example, when the object recognition unit 52 recognizes the detection area 91 as shown in FIG. 7 in the previous frame and fails to detect the detection area in the next frame, the tracking unit 53 recognizes a partial image similar to the area image in the detection area 91 shown in FIG. 7 (shown as area image 201 in FIG. 8) from the image of the next frame (image 22 in FIG. 8). If the tracking unit 53 recognizes a partial image similar to the region-inside image 201 in the image of the next frame (image 22 in FIG. 8), it outputs region 92 including the partial image as the detection region of the image of the next frame (image 22 in FIG. 8), and outputs the coordinate information (X1, Y1) and (X2, Y2) of each pixel in the upper left and lower right of region 92 as the coordinate information of the detection region.

In this embodiment, in order to deal with cases where an object that was detected in the previous frame is not detected in the next frame due to a small change in the posture or position of the background or the detection object, the tracking technology of image processing is applied to estimate the detection area of the next frame. In addition, when the previous image includes multiple detection areas, for example, the tracking unit 53 can track the detection result with the highest certainty detected in the previous frame. The certainty indicates the percentage probability that the recognition model 572 thinks that the detection object is a human, and roughly speaking, it is a value that indicates how confident the recognition model 572 is in the detection result. When an undetected frame appears, the tracking unit 53 searches for an area similar to the image in the area detected in the previous frame in the next frame (first frame). If the object recognition unit 52 has not detected the detection area in the next frame (second frame), the tracking unit 53 searches for a partial image in the second frame that is similar to the image in the area of the detection area estimated in the first frame. However, continuing tracking in response to a momentary false detection can result in unnecessarily expanding the false detection, so for example, a limit can be placed on the number of continuous tracking attempts.

When the coordinate information of a detection area including a person (target object) satisfies a predetermined exclusion condition, the recognition result filter 54 excludes the detection area that satisfies the condition. When the recognition result by the target object recognition unit 52 is obviously wrong, it is possible to reduce erroneous recognition by excluding it according to a rule. For example, when the position of a person in the captured image is limited due to the reflection of the vehicle body 101B due to the installation location of the imaging device 2 or the expected site conditions, the following filtering is possible for the area shown in FIG. 9 (detection areas 93, 94, and 97). Note that FIG. 9 shows a schematic example of a case where the target object recognition unit 52 recognizes detection areas 93 to 97 when an image 23 including people 82 and 83 is input to the target object recognition unit 52.

(a) If the coordinates of the feet (the coordinates at the bottom left of the detection area) are above the reference value indicated by the dashed line L1, it is determined to be an erroneous recognition. For example, this assumes a case where the recognition is at a height that is known to be sky. This is effective when there is no steep upward gradient in the range to be detected. In the example shown in Figure 9, the detection area 93 can be excluded.

(b) If the aspect ratio of the detection area is wider than a predetermined reference value, it is deemed to be an erroneous recognition. This is effective when the state of the detection target is limited to the following: (1) Standing. (2) Walking. (3) Crouching. (4) Most of the body is visible (not at an extremely close distance where only the head is visible). In the example shown in Figure 9, detection area 97 can be excluded.

(c) If the reflected area of the vehicle body 101B is covered by more than a certain amount, it is determined to be a false recognition. In the example shown in FIG. 9, the detection area 94 can be excluded.

The distance estimation unit 55 uses the distance estimation model 573 to estimate the distance to the person (object) as a distance acquisition value by inputting the image and the coordinate information output by the object recognition unit 52. The distance estimation model 573 is a trained model that, like the recognition model 572, uses, for example, a neural network as an element. The distance estimation model 573 is a model that inputs information representing an image and coordinate information of a detection area, estimates and outputs the distance to a person, and is machine-learned using a set of information representing an image, coordinate information of a detection area, and information representing the distance to the person as teacher data. The distance estimation unit 55 inputs, for example, information representing the image 21 shown in FIG. 6 and coordinate information (X1, Y1) and (X2, Y2) of the detection area 91 shown in FIG. 7, estimates the distance from the imaging device 2 to the person 81, and outputs it as a distance acquisition value.

In this embodiment, the distance estimation model 573 uses the coordinate information of the detection area as input data.

The coordinate conversion unit 56 converts the distance calculation value in the camera coordinate system based on the imaging device 2 estimated by the distance estimation unit 55 into a distance calculation value in the vehicle body coordinate system based on the vehicle body 101B.

The distance calculation unit 6 calculates a distance calculation value representing the latest distance to a person (target object) based on multiple distance acquisition values in a time series, for example by using a moving average.

The output unit 7 compares the distance calculation value calculated by the distance calculation unit 6 with a predetermined threshold value and outputs a predetermined signal based on the comparison result. When the distance calculation value falls below the predetermined threshold value, the output unit 7 outputs a predetermined signal to the output device 4, which is composed of a monitor, a speaker, an indicator light, etc., to output a warning display or a warning sound, or outputs a predetermined signal to a control device (not shown) of the hydraulic excavator 101 to limit or stop the operation of the hydraulic excavator 101 (control intervention). In this embodiment, the output unit 7 outputs a signal for issuing a warning or executing a control intervention, etc., when the distance calculation value is smaller than a predetermined threshold value (for example, 4 m). However, in other embodiments, the output unit 7 may output a signal for issuing a warning or executing a control intervention when the distance calculation value is greater than the predetermined threshold value (when the target object is too far away), or may output a signal for issuing a warning or executing a control intervention when the distance calculation value deviates from the range defined by the respective threshold values of the predetermined upper and lower limits (when the target object is too far away or too close).

As mentioned in the tracking section above, even minute changes between previous and next frames can change the recognition results, which could result in, for example, the control signal switching between ON and OFF drastically, causing problems with control intervention or alert issuance. Therefore, in this embodiment, the distance calculation unit 6 uses multiple distance acquisition values in the acquired time series to perform time filtering (smoothing) based on a time average to obtain a distance calculation value, thereby keeping the fluctuations in the distance calculation value lower than when time filtering is not performed.

The output device 4 is composed of a monitor, a speaker, an indicator light, etc., or a combination of these, and is provided, for example, in the driver's cab 104. Note that, for example, if the work machine to which the imaging device 2 is attached is remotely operated using a wireless signal, the monitoring device 3 and the output device 4 can be provided at the location where the remote operation is performed.

Next, an example of the operation of the monitoring device 3 shown in FIG. 1 will be described with reference to FIG. 4. The process shown in FIG. 4 is executed for each frame of video captured by the imaging device 2. When the process shown in FIG. 4 is executed, first, the input unit 51 inputs an image from the imaging device 2 (step S1). Next, the object recognition unit 52 recognizes a person included in the image (recognizes an object present in the target range) (step S2). In step S2, the object recognition unit 52 recognizes a person (object) from the image using the recognition model 572, and outputs coordinate information of the area (detection area) including the recognized person.

Next, if the person (target) is not recognized ("NO" in step S3), the tracking unit 53 performs tracking processing (step S4). In step S4, the tracking unit 53 regards the coordinate information of an area in the image in which the person (target) is not recognized that includes a partial image similar to an image within the area (detection area) including the person (target) in the previous image as the coordinate information of the area (detection area) including the person (target) in the image in which the person (target) is not recognized.

Next, if the coordinate information of an area (detection area) containing a recognized person (target object) satisfies a predetermined exclusion condition, the recognition result filter 54 excludes the area that satisfies the condition (step S5).

Next, the distance estimation unit 55 and the coordinate conversion unit 56 acquire (calculate) a distance acquisition value that indicates the distance to the person (target object) (step S6). In step S6, the distance estimation unit 55 uses the image and the coordinate information of the detection area as inputs to estimate the distance to the person (target object) as a distance acquisition value using the distance estimation model 573. In addition, the coordinate conversion unit 56 converts the distance acquisition value in the camera coordinate system based on the imaging device 2, estimated by the distance estimation unit 55, into a distance acquisition value in the vehicle body coordinate system based on the vehicle body 101B.

Next, the distance calculation unit 6 calculates a distance calculation value representing the latest distance to the person (target object) based on the multiple distance acquisition values in time series (step S7).

Next, the output unit 7 compares the calculated distance value with a predetermined threshold value (step S8) and outputs a predetermined signal based on the comparison result (step S9).

Next, the process of step S7 shown in FIG. 4 will be described with reference to FIG. 5. FIG. 5 shows an example of the process of step S7 shown in FIG. 4. In the process shown in FIG. 5, first, the distance calculation unit 6 stores the value of variable Z(i) in variable Z(i+1) while changing variable i by 1 at a time from N-1, ..., 2, 1 (step S71). Variable Z(i) is a variable that stores N distance acquisition values in a time series, variable Z(1) stores the latest distance acquisition value, variable Z(2) stores the distance acquisition value of one frame before, and variable Z(N) stores the distance acquisition value of the (N-1) frame before. Note that N is an integer of 2 or more.

Next, the distance calculation unit 6 determines whether a person (target object) is recognized (step S72), and if a person (target object) is recognized (if "YES" in step S72), obtains the new distance acquisition value obtained in step S6 (step S73). Note that it is determined in step S72 that a person (target object) is recognized when a person (target object) is recognized in step S2 (coordinate information of the detection area is calculated) or coordinate information of the detection area is set in step S4, and then coordinate information of the detection area that was not excluded in step S5 exists. Also, in step S73, if there are multiple detection areas in the target image, the distance calculation unit 6 sets the smallest distance acquisition value (the value with the smaller margin for the threshold) of the multiple distance acquisition values estimated for each detection area as the new distance acquisition value.

On the other hand, if a person (target object) is not recognized (step S72: NO), the distance calculation unit 6 fixes the new distance acquisition value to the first constant (step S74). In this embodiment, the first constant is the maximum value (limit value) of the distance acquisition value, and for example, if the threshold value is 4 m, the first constant can be set to a value greater than the threshold value (for example, 6 m).

Next, the distance calculation unit 6 inputs the new distance acquisition value and the value of the first constant that has the smaller margin relative to the threshold into variable Z(1) (step S75). Here, the smaller margin relative to the threshold means that when the threshold is the lower limit as in this embodiment (when a warning or the like is issued when the value is smaller than the threshold), when the value is greater than the threshold, the smaller the difference from the threshold, the smaller the margin. Also, when the value is smaller than the threshold, the larger the difference (absolute value) from the threshold, the smaller the margin. For example, when the threshold is 4m and the first constant is 6m, when the distance acquisition value is 7m, the first constant has the smaller margin relative to the threshold, and the first constant 6m is input into variable Z(i). Also, for example, when the distance acquisition value is 3m, the distance acquisition value has the smaller margin relative to the threshold, and the distance acquisition value 3m is input into variable Z(i).

Note that, unlike this embodiment, for example, if the threshold is an upper limit value (if a warning or the like is issued when the value is greater than the threshold), when the value is smaller than the threshold, the smaller the difference from the threshold, the smaller the margin. Also, when the value is greater than the threshold, the larger the difference (absolute value) from the threshold, the smaller the margin. Note that in this case, the first constant is the minimum value (limit value) of the distance acquisition value, and for example, when the threshold is 6 m, the first constant can be set to a value smaller than the threshold (for example, 4 m). For example, when the threshold is 6 m and the first constant is 4 m, when the distance acquisition value is 3 m, the first constant has a smaller margin relative to the threshold, and the first constant 4 m is input to variable Z (i). Also, for example, when the distance acquisition value is 7 m, the distance acquisition value has a smaller margin relative to the threshold, and the distance acquisition value 7 m is input to variable Z (i).

Next, when the distance calculation value is a variable D, the distance calculation unit 6 calculates the distance calculation value, for example, using the formula D = {Z(1) + Z(2) + ... + Z(N)}/N (step S76).

In the process shown in FIG. 5, the distance calculation unit 6 calculates the simple moving average of the latest distance acquisition value and the previous N-1 distance acquisition values (distance acquisition values for the previous (N-1) frames) as the distance calculation value. In addition, by limiting the maximum distance acquisition value to the first constant, it is possible to improve the responsiveness of the distance calculation value when the distance acquisition value changes suddenly, compared to the case where no limit is imposed.

Figure 10 shows the results of verifying the effect of time filtering. From the top, Figure 10 shows the time changes of the acquired distance value (dashed line) and the calculated distance value (solid line), the time changes of the judgment result (without a time filter), and the time changes of the judgment result (with a time filter). The judgment results are "0" when the value is not below the threshold of 4m and "1" when the value is below the threshold of 4m. As shown in Figure 10, the frequency with which the judgment result changes is greater when there is no time filter (when the acquired distance value is compared with the threshold) than when there is a time filter (when the calculated distance value is obtained using a moving average).

As described above, according to this embodiment, a distance calculation value that represents the latest distance to a person (target object) is calculated based on multiple distance acquisition values in a time series, so that distance-based determination processing can be performed stably.

Next, a modified example of the above embodiment will be described with reference to Figs. 11 to 14. This modified example differs from the above embodiment in the way in which the distance calculation value is calculated by the distance calculation unit 6 in step S7 shown in Fig. 4 and in the operation of the output unit 7. Figs. 11 to 13 are flow charts showing another example of the processing of step S7 shown in Fig. 4, and the flows in each figure are connected to each other by connectors C1 and C2. Fig. 14 is a schematic diagram for explaining another example of the processing of step S7 shown in Fig. 4 and shown in Figs. 11 to 13. Note that the configuration of the monitoring system 1 shown in Fig. 1 and the operation example shown in Fig. 4 are basically the same as the above embodiment and this modified example, except for the way in which the distance calculation value is calculated and the operation of the output unit 7.

In this modified example, the distance acquisition unit 5 outputs the coordinate information of the detection area together with the acquired distance acquisition value to the distance calculation unit 6. That is, the distance acquisition unit 5 recognizes a person (object) present in the target range, acquires a distance acquisition value representing the distance to the recognized person (object), and further acquires information representing the position of the person (object) (coordinate information of the detection area). Then, the distance calculation unit 6 selectively associates the distance acquisition value with a plurality of divided areas, which are areas obtained by dividing the target range into a plurality of areas, based on the position (coordinate information of the detection area), and calculates the distance calculation value of each divided area based on the plurality of distance acquisition values in time series associated with each divided area and the distance calculation value of other adjacent divided areas for each divided area. In addition, for divided areas in which a person (object) is not recognized, the distance calculation unit 6 fixes the plurality of distance acquisition values in time series to a predetermined constant (second constant) based on the distance calculation value of other adjacent divided areas, and calculates the distance calculation value.

As shown in FIG. 14, the distance calculation unit 6 calculates distance calculation values for each of three divided regions obtained by dividing the image 24 input by the input unit 51 horizontally (perpendicular to the rotation axis of the upper rotating body 103). Hereinafter, the divided region on the left side shown in FIG. 14 is referred to as the left region 2L, the divided region in the middle as the middle region 2C, and the divided region on the right side as the right region 2R. Note that the image 24 shown in FIG. 14 includes a person 84 and a person 85, and the distance acquisition unit 5 recognizes a detection region 98 for the person 84 and a detection region 99 for the person 85. Furthermore, the detection regions 98 and 99 are located in the right region 2R.

Now, the process of step S7 shown in FIG. 4 will be described with reference to FIG. 11 to FIG. 13. In the process shown in FIG. 11 to FIG. 13, first, the distance calculation unit 6 stores the value of the variable Zl(i) in the variable Zl(i+1) while changing the variable i by 1 at a time from N-1, ..., 2, 1 (step S701). The variable Zl(i) is a variable that stores N distance acquisition values in a time series associated with the left region 2L, the latest distance acquisition value associated with the left region 2L is stored in the variable Zl(1), the distance acquisition value of the previous frame associated with the left region 2L is stored in the variable Zl(2), and the distance acquisition value of the (N-1) frame ago associated with the left region 2L is stored in the variable Zl(N). Note that N is an integer of 2 or more.

The distance calculation unit 6 then changes the variable i by 1 each time from N-1, ..., 2, 1, and stores the value of the variable Zc(i) in the variable Zc(i+1) (step S702). The variable Zc(i) is a variable that stores N distance acquisition values in a time series associated with the medium region 2C, the latest distance acquisition value associated with the medium region 2C is stored in the variable Zc(1), the distance acquisition value of the previous frame associated with the medium region 2C is stored in the variable Zc(2), and the distance acquisition value of the (N-1) frame ago associated with the medium region 2C is stored in the variable Zc(N).

The distance calculation unit 6 then changes the variable i by 1 each time from N-1, ..., 2, 1, and stores the value of the variable Zr(i) in the variable Zr(i+1) (step S703). The variable Zr(i) is a variable that stores N distance acquisition values in a time series associated with the right region 2R, the latest distance acquisition value associated with the right region 2R is stored in the variable Zr(1), the distance acquisition value of the previous frame associated with the right region 2R is stored in the variable Zr(2), and the distance acquisition value of the previous (N-1) frame associated with the middle region 2C is stored in the variable Zc(N).

The distance calculation unit 6 then selects a divided area to which the new distance acquisition value is to be associated based on the coordinate information of the area (detection area) that includes the recognized person (target object) (step S704). In step S704, for example, if the detection area is included in one divided area, the distance calculation unit 6 associates the new distance acquisition value estimated for the detection area with that one divided area. For example, if the detection area spans two divided areas, the distance calculation unit 6 associates the new distance acquisition value estimated for the detection area with the two divided areas.

The distance calculation unit 6 then stores the new distance acquisition value in the variable Zl(1), Zc(1), or Zr(1) corresponding to the selected divided region (step S705). However, in step S704, if multiple detection regions are associated with one divided region, the smallest distance acquisition value (the value with the smaller margin to the threshold) among the multiple distance acquisition values estimated for each detection region is set as the new distance acquisition value. In addition, the new distance acquisition value is set as the value between the distance acquisition value and the first constant that has the smaller margin to the threshold.

Next, the distance calculation unit 6 stores the first constant in the variable Zl(1), Zc(1), or Zr(1) corresponding to the unselected divided area (step S706).

In the case of image 24 shown in FIG. 14, the distance calculation unit 6 selects the right region 2R in step S704, stores the distance acquisition value estimated for, for example, detection region 99 in variable Zr(1) in step S704, and stores a first constant (6 m) in variables Zl(1) and Zc(1) in step S705.

Next, when the calculated distance value of the left region 2L is the variable Dl, the distance calculation unit 6 calculates the calculated distance value of the left region 2L using the formula Dl = {Zl(1) + Zl(2) + ... + Zl(N)}/N (step S707).

Next, when the calculated distance value of the middle region 2C is the variable Dc, the distance calculation unit 6 calculates the calculated distance value of the middle region 2C using the formula Dc = {Zc(1) + Zc(2) + ... + Zc(N)}/N (step S708).

Next, when the calculated distance value of the right region 2R is the variable Dr, the distance calculation unit 6 calculates the calculated distance value of the right region 2R using the formula Dr = {Zr(1) + Zr(2) + ... + Zr(N)}/N (step S709).

Next, the distance calculation unit 6 determines whether the distance calculation value Dl of the left region 2L satisfies a first condition and the distance calculation value Dc of the adjacent middle region 2C satisfies a second condition (step S710). For example, the first condition is that the distance calculation value Dl is not below a threshold value (is not a value that outputs a signal for a warning, etc.), and the second condition is that the distance calculation value Dc is below a threshold value (is a value that outputs a signal for a warning, etc.).

If the distance calculation value Dl of the left region 2L satisfies the first condition and the distance calculation value Dc of the adjacent middle region 2C satisfies the second condition (if "YES" in step S710), the distance calculation unit 6 fixes the variable Zl(i) (i = N-1, ..., 2) whose margin to the threshold is greater than the second constant to the second constant (step S711), and then recalculates the distance calculation value Dl of the left region 2L in the same manner as in step S707 (step S712). The second constant is a fixed value whose margin to the threshold is smaller than the first constant. For example, if the threshold (lower limit) is 4m and the first constant is 6m, the second constant can be a value greater than 4m and less than 6m. For example, if the second constant is 4.2m, in step S711, the value of the variable Zl(i) greater than 4.2m is fixed to 4.2m. By doing this, for example, when a person (object) that caused the distance calculation value Dc in the middle region 2C to fall below the threshold moves to the left region 2L, the distance calculation value Dl in the left region 2L can be made to fall below the threshold with a smaller delay than when the second constant is not fixed. Note that the first and second conditions are not limited to those described above. Also, the first condition may be omitted.

The distance calculation unit 6 then determines whether the distance calculation value Dc of the middle region 2C satisfies the first condition, and whether the distance calculation value Dl of the adjacent left region 2L and the distance calculation value Dr of the adjacent right region 2R satisfy the second condition (step S713). For example, the first condition is that the distance calculation value Dc is not below a threshold value (is not a value that outputs a signal for a warning, etc.), and the second condition is that the distance calculation value Dl or the distance calculation value Dr is below a threshold value (at least one of them is a value that outputs a signal for a warning, etc.).

If the distance calculation value Dc for the middle region 2C satisfies the first condition and the distance calculation value Dl for the adjacent left region 2L and the distance calculation value Dr for the right region 2R satisfy the second condition (if "YES" in step S713), the distance calculation unit 6 fixes the variable Zc(i) (i = N-1, ..., 2) whose margin to the threshold is greater than the second constant to the second constant (step S714), and then recalculates the distance calculation value Dc for the middle region 2C in the same manner as in step S708 (step S715).

Next, the distance calculation unit 6 determines whether the distance calculation value Dr for the right region 2R satisfies the first condition and the distance calculation value Dc for the adjacent middle region 2C satisfies the second condition (step S716). For example, the first condition is that the distance calculation value Dr is not below a threshold value (is not a value that outputs a signal for a warning, etc.), and the second condition is that the distance calculation value Dc is below a threshold value (is a value that outputs a signal for a warning, etc.).

If the distance calculation value Dr for the right region 2R satisfies the first condition and the distance calculation value Dc for the adjacent middle region 2C satisfies the second condition (YES in step S716), the distance calculation unit 6 fixes the variable Zr(i) (i=N-1, ..., 2) whose margin to the threshold is greater than the second constant to the second constant (step S717), and then recalculates the distance calculation value Dc for the middle region 2C in the same manner as in step S708 (step S716).

In addition, in this modified example, the output unit 7 compares the calculated distance value for each divided area with a threshold value, and outputs a predetermined signal for each divided area or for all areas in common depending on the comparison result. For example, when the output device 4 is a monitor, the output unit 7 can display a warning or the like so that a divided area that falls below the threshold can be visually identified, can illuminate a warning light at a position corresponding to the divided area, and can switch the restriction on the rotation direction of the upper rotating body 103 depending on the direction of the divided area.

In the process shown in Figs. 11 to 13, the distance calculation unit 6 calculates, for each divided region, the simple moving average of the latest distance acquisition value and the previous N-1 distance acquisition values (distance acquisition values for the previous (N-1) frames) as the distance calculation value. Also, by limiting the maximum distance acquisition value to a first constant, it is possible to improve the responsiveness of the distance calculation value when the distance acquisition value changes suddenly compared to when no limitation is imposed. Furthermore, by limiting the maximum distance acquisition value of the region to a second constant smaller than the first constant based on the distance acquisition value of an adjacent divided region, for example when the distance acquisition value of the adjacent divided region is below a threshold, it is possible to improve the responsiveness of the distance calculation value when the distance acquisition value changes suddenly compared to when no limitation is imposed.

Although the embodiment of the present invention has been described above with reference to the drawings, the specific configuration is not limited to the above embodiment, and design changes within the scope of the present invention are also included. For example, the work machine is not limited to a hydraulic excavator. In addition, when the imaging device 2 is configured with multiple cameras as shown in FIG. 3, processing of multiple captured images may be performed independently in parallel, or multiple captured images may be processed as multiple divided regions. In addition, the moving average in the time filter is not limited to a simple moving average, but may be a weighted moving average, an exponential moving average, or the like. In addition, the time filter may be a low-pass filter consisting of a finite impulse response filter, an infinite impulse response filter, or the like. In addition, the imaging device may be installed on a moving body, or may be fixed to the ground, for example.

In addition, some or all of the programs executed by the computer in the above embodiments can be distributed via computer-readable recording media or communication lines.

1...surveillance system, 2...imaging device, 3...surveillance device, 4...output device, 5...distance acquisition unit, 6...distance calculation unit, 7...output unit, 2L...left region (divided region), 2C...middle region (divided region), 2R...right region (divided region), 51...input unit, 52...object recognition unit, 53...tracking unit, 54...recognition result filter, 55...distance estimation unit, 572...recognition model, 573...distance estimation model

Claims (10)

a distance acquisition unit that recognizes an object present in a target range and acquires a distance acquisition value that indicates a distance to the recognized object;
a distance calculation unit that calculates a distance calculated value representing a latest distance to the object based on a plurality of the distance acquisition values in time series;
an output unit that compares the calculated distance value with a predetermined threshold value and outputs a predetermined signal based on a comparison result;
Equipped with
when the object is not recognized, the distance calculation unit fixes the plurality of distance acquisition values in time series to a predetermined constant and calculates the distance calculation value ;
When the threshold value is a lower limit value, the predetermined constant is set as a value greater than the threshold value and as a maximum value of the distance acquisition value;
When the threshold value is an upper limit value, the predetermined constant is a value smaller than the threshold value and set as a minimum value of the distance acquisition value.
Work machine monitoring device. The distance acquisition unit further acquires information representing a position of the object,
2. The monitoring device for a work machine according to claim 1, wherein the distance calculation unit selectively associates the distance acquisition value with a plurality of divided areas obtained by dividing the target range into a plurality of areas based on the position, and calculates the distance calculation value for each of the divided areas based on a plurality of distance acquisition values in a time series associated with each of the divided areas. The monitoring device for a work machine according to claim 2 , wherein the distance calculation unit further calculates the distance calculation value for each of the divided regions based on the distance calculation values for the other adjacent divided regions. The monitoring device for a work machine according to claim 3 , wherein the distance calculation unit calculates the distance calculation value for the divided area in which the object is not recognized by fixing a plurality of the distance acquisition values in a time series to a predetermined constant based on the distance calculation values for other adjacent divided areas. The distance acquisition unit,
an input unit for inputting an image signal representing an image captured of the target range;
an object recognition unit that recognizes the object from the image and outputs coordinate information that indicates a position of an area including the recognized object;
a storage unit that stores a distance estimation model that receives input of information representing the image and the coordinate information, estimates a distance to the object, and outputs the distance estimation model, the distance estimation model being trained using a set of information representing the image, the coordinate information, and information representing the distance to the object as training data;
a distance estimation unit that estimates a distance to the object as the distance acquisition value by using the distance estimation model and receiving as input the image and the coordinate information output by the object recognition unit;
having
A monitoring device for a work machine according to any one of claims 1 to 4. The distance acquisition unit,
a tracking unit which, when the object recognition unit does not recognize the object, sets, in the image in which the object is not recognized, coordinate information of an area including a partial image similar to an image in an area including the object in the previous image, as coordinate information of an area including the object in the image in which the object is not recognized.
A monitoring device for a work machine according to claim 5. The distance acquisition unit,
a recognition result filter for excluding an area that satisfies a predetermined exclusion condition when the coordinate information of the area including the object satisfies the predetermined exclusion condition.
A monitoring device for a work machine according to claim 5 or 6. A step of recognizing an object present in a target range and acquiring a distance acquisition value representing a distance to the recognized object;
calculating a distance calculation value representing a latest distance to the object based on a plurality of the distance acquisition values in time series;
comparing the calculated distance value with a predetermined threshold value and outputting a predetermined signal based on a comparison result;
Including,
In the step of calculating the distance calculation value, if the object is not recognized, the plurality of distance acquisition values in time series are fixed to a predetermined constant to calculate the distance calculation value ;
When the threshold value is a lower limit value, the predetermined constant is set as a value greater than the threshold value and as a maximum value of the distance acquisition value;
When the threshold value is an upper limit value, the predetermined constant is a value smaller than the threshold value and set as a minimum value of the distance acquisition value.
A method for monitoring a work machine. a distance acquisition unit that recognizes an object present in a target range and acquires a distance acquisition value that indicates a distance to the recognized object;
a distance calculation unit that calculates a distance calculated value representing a latest distance to the object based on a plurality of the distance acquisition values in time series;
an output unit that compares the distance calculation value with a predetermined threshold and outputs a predetermined signal based on a comparison result;
The distance acquisition unit further acquires information representing a position of the object,
the distance calculation unit selectively associates the distance acquisition value with a plurality of divided regions obtained by dividing the target range into a plurality of regions based on the position, and calculates the distance calculation value for each of the divided regions based on the plurality of distance acquisition values in a time series associated with each of the divided regions;
further calculating the distance calculation value of each of the divided regions based on the distance calculation values of the other divided regions adjacent to each other;
Work machine monitoring device. a distance acquisition unit that recognizes an object present in a target range and acquires a distance acquisition value that indicates a distance to the recognized object;
a distance calculation unit that calculates a distance calculated value representing a latest distance to the object based on a plurality of the distance acquisition values in time series;
an output unit that compares the distance calculation value with a predetermined threshold and outputs a predetermined signal based on a comparison result;
The distance acquisition unit,
an input unit for inputting an image signal representing an image captured of the target range;
an object recognition unit that recognizes the object from the image and outputs coordinate information that indicates a position of an area including the recognized object;
a storage unit that stores a distance estimation model that receives input of information representing the image and the coordinate information, estimates a distance to the object, and outputs the distance estimation model, the distance estimation model being trained using a set of information representing the image, the coordinate information, and information representing the distance to the object as training data;
a distance estimation unit that estimates a distance to the object as the distance acquisition value by using the distance estimation model and receiving the image and the coordinate information output by the object recognition unit as inputs.
Work machine monitoring device.

Images