JP7633508B2 - System for detecting position of object around work machine, and program for detecting position of object around work machine - Google Patents

Description

The present invention relates to a system for detecting objects around a work machine such as a construction machine, and a program for detecting the position of an object around the work machine.

For example, in a work machine (construction machine) with a structure specialized for various tasks, such as a hydraulic excavator with a lower traveling body and an upper rotating body that is rotatably attached to the lower traveling body, an operator can ride in the control room and operate a travel control part such as an operating lever to move the vehicle forward, backward, or turn. When performing such operations, the operator usually grasps the situation around the vehicle by visually checking or by viewing images taken by an on-board camera installed at an appropriate location on the vehicle on a monitor installed in the cab, and confirms that there are no people or obstacles in the direction the vehicle is moving.

In addition, in order to prevent accidents, it is also important to check surrounding conditions that are invisible or difficult to see by displaying camera images on a monitor and to accurately grasp the position of objects (detection targets) in the images displayed on the monitor. Therefore, it is possible to configure the work machine to be equipped with a sensor that measures the distance to the detection target captured by the camera, but implementing a sensor dedicated to distance measurement directly leads to increased costs.

Meanwhile, an increasing proportion of vehicles, such as ordinary passenger cars, are equipped with equipment (active safety) for preventing traffic accidents. One example of active safety is a technology that warns the driver of obstacles around the vehicle by using distance measuring devices such as laser radar, or image processing technology that analyzes images from an on-board camera (for example, see Patent Document 1 below).

JP 2011-48485 A

However, as mentioned above, when considering the cost and the size and performance of the controller installed in the work machine, it is difficult to apply such active safety technology to work machines.

The applicant is therefore developing a system that can detect the position of a detection target captured by an imaging device with high accuracy even in work machines that do not have a dedicated sensor or laser radar for measuring distance. Specifically, the system has a function for calculating the position of a detection target based on the results of a weighting process in which the coordinates of the detection frame when a person captured by the imaging device is image-recognized are used as weights, and even if the image recognition results vary, it is possible to avoid a situation in which the variation in the image recognition results is directly reflected in the calculation result. The system detects the position of a detection target with high accuracy and can calculate the distance from the imaging device to the detection target with high accuracy. The applicant is working to put the system into practical use.

However, when two people are detected in an image, for example when it is desired to detect the position of the person relatively closer to the imaging device, the coordinates of the detection frame of the person relatively farther from the imaging device are also used in the weighting process, which may result in an error in the detection value of the position of the person closer to the imaging device, and a position farther away than the actual distance may be calculated as the detected position of the person closer to the imaging device.

The present invention was made with a focus on these problems, and its main objective is to provide a surrounding detection object position detection system that can be applied to a work machine, a system that can detect the position of a detection object captured by an imaging device with high accuracy even in a work machine that does not have a dedicated sensor or laser radar for measuring distance, and further, a system that can calculate the distance from the imaging device to any one of the detection objects with high accuracy even when multiple detection objects are detected in the captured image.

In other words, the work machine surroundings monitoring system of the present invention, which is capable of detecting the position of a detection object around a work machine, comprises a detection unit that detects candidate positions of a detection object in an image captured by an imaging device attached to the work machine by using detection frames of multiple sizes for the image, a detection area limiting unit that limits the detection area for a priority detection object that is detected with priority based on the detection result by the detection unit to within a specified area in the image, and a detection coordinate calculation unit that calculates the position of the priority detection object as detection coordinates based on the detection result by the detection unit within the limited detection area limited by the detection area limiting unit. The work machine periphery monitoring system according to the present invention is characterized by using, as the detection area limiting unit, a reference coordinate calculation unit that calculates reference coordinates that serve as the reference for the limited detection area for a reference detection frame that serves as a reference among multiple detection frames in the image based on the detection results by the detection unit, a limited detection area calculation unit that calculates a limited detection area based on the reference coordinates, a comparison coordinate calculation unit that calculates comparison coordinates for each of the non-reference detection frames that are detection frames other than the reference detection frame among the multiple detection frames in the image based on the detection results by the detection unit, and a priority detection frame detection unit that extracts, from among the non-reference detection frames, detection frames whose comparison coordinates calculated by the comparison coordinate calculation unit are within the limited detection area as priority detection frames.

In this way, with the work machine surroundings detection object position detection system according to the present invention, even if multiple or more detection objects are detected using a detection frame, one of the detection objects is set as a priority object, and the detection frame related to the priority detection object is extracted, so that the detection frames related to detection objects other than the priority detection object are not treated as valid data when calculating the position of the priority detection object as a detection coordinate, and the position of the priority detection object captured by the imaging device can be detected with high accuracy. Specifically, with the work machine surroundings detection object position detection system according to the present invention, a reference detection frame that serves as a reference among multiple detection frames in the image is used to calculate reference coordinates that serve as the reference for the limited detection area, and the limited detection area is calculated based on the reference coordinates, so that the limited detection area can be set in the detection object image with a relatively simple calculation process. In addition, in the detection system for detecting the position of a detection object around a work machine according to the present invention, comparison coordinates are calculated for each of the non-reference detection frames, which are detection frames other than the reference detection frame, among the multiple detection frames in the image, and when calculating the position of the priority detection object, only the priority detection frames having comparison coordinates within the limited detection area are used together with the reference detection frame, so that the position of the priority detection object can be detected without being affected by the coordinate values of the non-reference detection frames for detection objects other than the priority detection object.

A suitable example of the "reference coordinates of the reference detection frame" in the present invention is the central X coordinate of the reference detection frame. It is also preferable that the "comparison coordinates of the non-reference detection frame" are similar to the "reference coordinates of the reference detection frame", and a suitable example of the comparison coordinates is the central X coordinate of the non-reference detection frame.

In addition, in the work machine periphery monitoring system according to the present invention, if the detection coordinate calculation unit calculates the position of the priority detection object in the detection target image as detection coordinates based on the processing results of weighting according to the count number (density number) of detection object specific coordinates that can identify the position of the priority detection object from the reference detection frame and detection object specific coordinates that can identify the position of the priority detection object from the priority detection frame having comparison coordinates within the limited detection area for each region into which the detection target image in the video is divided, then even if the detection results using the detection frame vary, the variation can be reduced by weighting processing that increases the weight as the count number increases, and the detection coordinates that identify the position of the priority detection object in the detection target image can be calculated with high accuracy. Note that when a single detection object is detected by the detection unit in the video, the detection object becomes the priority detection object.

A preferred configuration of the weighting processing unit in the present invention includes a large area weighting processing unit that performs weighting for each large area obtained by dividing the entire detection target image into multiple areas according to the number of counts of the detection target object identification coordinates, and a small area weighting processing unit that performs weighting for each small area obtained by further dividing each large area into multiple areas according to the number of counts of the detection target object identification coordinates. In this case, if the detection coordinates are calculated based on the processing results by the large area weighting processing unit and the small area weighting processing unit, the detection target image can be divided into smaller areas, and the detection coordinates can be calculated with high accuracy that further reflects the density of the detection target object identification coordinates.

A specific example of weighting in the weighting processing unit is weighted average processing based on the count number of specific coordinates of the detected object.

When detecting a detection target in the vicinity of a work machine, it is sufficient to detect a detection target (person) in a standing posture, and in particular to identify the position of the person's feet. Therefore, in the present invention, the coordinates corresponding to the bottom side of the rectangular detection frame are set as the priority detection target identification coordinates.

The present invention also includes a configuration for quickly calculating the real-time detection coordinate as the priority detection coordinate, but in order to calculate the detection coordinate considering the time series movement of the same detection object in the detection target image, it is preferable to configure it so that the final detection coordinate is calculated by taking the moving average of the real-time detection coordinate and multiple detection coordinates including the most recent past. In particular, when the real-time detection coordinate is in a close position within a specified range from the work machine, that is, when there is a person near the work machine, a quick response is required. Therefore, by adopting a configuration for changing the weight of the moving average according to the real-time detection coordinate (dynamically switching the weight), it becomes possible to immediately output the calculation result of a dangerous situation where a person is in a close position within a specified range from the work machine.

If the detection system for detecting the position of a detection target around a work machine according to the present invention is configured to measure the distance from the imaging device to the priority detection target based on the calculated detection coordinates, it becomes possible to calculate the distance to the priority detection target with high accuracy based on the position of the priority detection target in the target image, which is a calculation result that reduces the variability in the image recognition results using the detection frame (specifically, the position of the bottom edge of the detection frame).

Further, a work machine surroundings detection object position detection program according to the present invention is a software program executed by a computer as a work machine surroundings detection object position detection system for detecting the position of a detection object around the work machine, and includes a detection step for detecting position candidates of a detection object in an image captured by an imaging device attached to the work machine using detection frames of multiple sizes, a detection area limiting step for limiting a detection area for a priority detection object to be detected with priority based on a detection result by the detection step, to within a predetermined area in the image, and a calculation step for calculating a position of the priority detection object in the image as detection coordinates based on the detection result by the detection step within the limited detection area limited in the detection area limiting step. the detection area limiting step includes a reference coordinate calculation step of calculating reference coordinates that serve as the reference of the limited detection area for a reference detection frame that serves as a reference among the multiple detection frames in the image based on the detection result by the detection step; a limited detection area calculation step of calculating a limited detection area based on the reference coordinates; a comparison coordinate calculation step of calculating comparison coordinates for non-reference detection frames that are detection frames other than the reference detection frame among the multiple detection frames in the image based on the detection result by the detection step ; and a priority detection frame detection step of extracting, from the non-reference detection frames, detection frames whose comparison coordinates calculated in the comparison coordinate calculation step are within the limited detection area as priority detection frames.

This program for detecting the position of a detection object around a work machine can exclude detection results (detection frame coordinates) related to detection objects other than the priority detection object when calculating the position of the priority detection object, and can calculate the position of the priority detection object with high accuracy.

According to the present invention, even if two or more detection objects are detected in multiple captured images, only the coordinate values of the detection frame that contributed to the detection of the priority detection object can be used as valid data, and a detection object position detection system around a work machine that can detect the position of the priority detection object with high accuracy can be provided. By applying this invention to a work machine that does not have a dedicated sensor or laser radar, it is possible to identify the position of a detection object around the work machine with high accuracy while avoiding high costs, making it a highly practical technology.

1 is a diagram illustrating an example of application of a system for detecting the position of a detection target around a work machine according to an embodiment of the present invention; 2 is an overall configuration diagram of a detection system for detecting the position of a detection target around a work machine according to the embodiment; 13 is a diagram showing an example of a detection object image immediately after execution of a detection coordinate calculation process in the embodiment; FIG. 5A and 5B are diagrams illustrating processing performed by a detection unit on a detection target image in the embodiment. 5 is a flowchart showing a detection program for detecting a position of a detection target around a work machine according to the embodiment; 5A and 5B are diagrams illustrating processing contents of a reference coordinate calculation unit for a detection target image in the embodiment. 5A and 5B are diagrams illustrating processing performed by a limited detection area calculation unit on a detection target image in the embodiment. 5A and 5B are diagrams illustrating processing performed by a comparison coordinate calculation unit for a detection target image in the embodiment. 4 is a diagram corresponding to FIG. 3 and showing an image in which accurate detection coordinates cannot be calculated in the embodiment; FIG. 5A to 5C are diagrams for explaining the processing content of a large area weighting unit in the embodiment. 5A to 5C are diagrams for explaining the processing content of a small region weighting unit in the embodiment. 5 is a diagram showing an equation used in a detection coordinate calculation process in the embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
As shown in FIG. 1, the detection system X for detecting a position of a detected object around a work machine according to this embodiment can be used to monitor the periphery of a work machine H, such as a hydraulic excavator (also called a backhoe). The detection system X for detecting a position of a detected object around a work machine includes a controller C capable of capturing an image of an imaging device I attached to the work machine H and outputting the image to an output destination. The imaging device I used in this embodiment is a general vehicle-mounted camera. The position at which the imaging device I is attached to the work machine H is not particularly limited as long as it is a position at which it can capture an image of a range (blind spot) or a range that is difficult to see from an operator who is in the cockpit H1. In this embodiment, the imaging device I is attached to a position at which it can capture an image of a predetermined range behind the hydraulic excavator H. Note that an imaging device that can provide a 360-degree bird's-eye view of the periphery of the work machine from the sky above the operator's cabin can also be applied. In FIG. 1, the devices (the imaging device I and the display unit D described later) used in implementing the detection system X for detecting a detected object around a work machine are each exaggerated and typically shown.

As shown in Figs. 1 and 2, the controller C includes a detection unit C1 that detects position candidates of a detection object in an image captured by the imaging device I using a detection frame F (see Fig. 3), a detection area limiting unit C2 that limits position candidates for a priority detection object to be detected on the basis of the detection result by the detection unit C1, and a detection coordinate calculation unit C3 that calculates the position of the priority detection object as detection coordinates on the basis of the position candidates limited (specified) by the detection area limiting unit C2.

The detection object in this embodiment is a person W in the vicinity of the work machine. The imaging device I captures a video (image) including the detection object in the environment surrounding the work machine H and imports it as image data. The detection object image V (video screen) detected by the detection unit C1 using the detection frame F may be the image data itself (unprocessed raw data), or it may be image data whose data size has been reduced for processing. Figure 3 shows an example of the detection object image V immediately after execution of the detection coordinate calculation step S3 described below.

As shown in FIG. 4, for example, the detection unit C1 uses a rectangular detection frame F to scan the entire detection target image V without missing anything, and detects the area where the person W exists. The size of the detection frame F in the detection unit C1 may be changed depending on the expected height, body type, etc. of the person W to be detected. In this embodiment, for example, HOG (Histograms of Oriented Gradients) is applied as the feature amount used to detect the person W in the detection target image, and SVM (Support Vector Machine) is applied as the learning method used for learning the detection.

The detection logic (detection step S1; see FIG. 5) of the detection unit C1 in this embodiment is as follows. Note that FIG. 5 is a flowchart of a detection program for detecting the position of a detected object around a work machine, which is a software program that causes a computer mounted on the work machine H to execute the detection program as a detection system X for detecting the position of a detected object around the work machine. This computer is either one provided as a component of the controller C or the controller C itself.

First, the detection target image V at the start of the detection process is searched with the detection frame F to detect position candidates of the person W, which is the detection target. Next, the detection target image V is reduced by a specified ratio, and the same size detection frame F is used to search again to detect position candidates of the person W. Then, the detection target image V is reduced a specified number of times, or the detection process ends when the detection target image V becomes smaller than the detection frame F. By performing the above processing, multiple position candidates are detected for each person W, and the position and size of the detection frame F in the detection target image V are different for each candidate. Figure 4 shows a schematic diagram of the detection target image V when multiple position candidates are detected for the same person W (multiple detection frames F are displayed). As shown in the figure, the position candidates of the person W may be detected with a detection frame F that is closely attached to the person W (a detection frame F of the same size as the image of the person W), or may be detected with a frame one size larger than the detection frame that is closely attached to the person W.

Also, as shown in Figure 4, when two or more people (two people W1 and W2 in the figure) are shown in the detection target image V, multiple position candidates are detected for each of the people W1 and W2, and the position and size of the detection frame F in the detection target image V differs for each of the people W1 and W2. The figure shows a state in which position candidates based on four detection frames F are detected for each of the people W1 and W2 (a total of eight position candidates based on detection frames F).

In this embodiment, the person W1 closer to the imaging device I is set as a detection target (priority detection target) that is given priority over the person W2 that is further away. Therefore, it is desirable that the position candidates based on the four detection frames F for the person W2 that is farther away from the imaging device I do not affect (are excluded from) calculation of the coordinates of the person W1, which is the priority detection target.

Therefore, in the work machine surrounding detection object position detection system X according to this embodiment, the detection area for the priority detection object is limited to a predetermined area in the image V by the detection area limiting unit C2 based on the detection result by the detection unit C1 (detection area limiting step S2; see FIG. 5).

The detection area limiting unit C2 includes a reference coordinate calculation unit C21, a limited detection area calculation unit C22, a comparison coordinate calculation unit C23, and a priority detection frame extraction unit C24.

As shown in Fig. 6, the reference coordinate calculation unit C21 specifies a reference detection frame FA that is a reference among the multiple detection frames F in the video V based on the detection result by the detection unit C1, and calculates reference coordinates that are the reference of the limited detection area for the reference detection frame FA. In this embodiment in which the person W1 closest to the imaging device I is set as the priority detection object, the reference coordinate calculation unit C21 sets the detection frame F that is closest to the imaging device I (the detection frame F shown by a relatively thick line in Fig. 6, which is the detection frame F that is the foreground in the figure (the detection frame F at the bottom in the video V)) among the position candidates of the multiple detection frames F in the detection target image V as the reference detection frame FA, and calculates reference coordinates that are the reference of the limited detection area for the reference detection frame FA (reference coordinate calculation step S21). As a specific calculation process in the reference coordinate calculation unit C21, in this embodiment, a process is applied in which the center X coordinate of the reference detection frame FA (the X coordinate shown by the straight line with the symbol "FSa" in Fig. 6) is calculated as the reference coordinate by the following formula (11).
Center X coordinate of reference detection frame=X coordinate of reference detection frame+detection frame width of reference detection frame×0.5 Equation (11)

The position of the detection frame F within the detection target image V can be found as coordinate values by a specified calculation. Therefore, the coordinate size of the detection frame F can be identified from the detection results by the detection unit C1, and the position coordinates (X coordinate) and detection frame width (length of the detection frame in the X-axis direction) of all detection frames F including the reference detection frame FA can also be identified. In this embodiment, the position of the bottom left corner (bottom left edge) of the rectangular detection frame F is set as the X coordinate of the detection frame F. Also, the bottom left corner of the video screen (detection target image V) is set as the origin of the XY coordinates.

The limited detection area calculation unit C22 calculates the limited detection area based on the coordinates of the reference detection frame FA. In this embodiment, the minimum and maximum X coordinates of the detection area are calculated by the following formulas (12) and (13) based on the X coordinate and detection frame width of the reference detection frame FA, and the area between the minimum and maximum X coordinates is calculated as the limited detection area E as shown in Fig. 7 (limited detection area calculation step S22).
Minimum X coordinate of detection area=X coordinate of reference detection frame+width of reference detection frame×0.25 (Formula (12))
Maximum X coordinate of detection area=X coordinate of reference detection frame+detection frame width of reference detection frame×0.75 (Equation (13))

In FIG. 7, the minimum and maximum X coordinates of the detection area are indicated by dashed lines. Through this process, as shown in FIG. 7, the area that is equal to or greater than the minimum X coordinate of the detection area and equal to or less than the maximum X coordinate of the detection area (area shaded in gray), that is, the area within a range of a predetermined dimension in the positive and negative directions of the X axis from the center X coordinate FSa of the reference detection frame FA, is defined as the limited detection area E.

The comparison coordinate calculation unit C23 calculates comparison coordinates for each of the non-reference detection frames F, which are detection frames F other than the reference detection frame FA, among the multiple detection frames F in the video V, based on the detection results by the detection unit C1. In this embodiment, for the detection frames F other than the reference detection frame FA among the detection frames F (seven detection frames F in total), the process calculates the center X coordinate of each detection frame F (the X coordinate indicated by the dashed line labeled with "FSb" in FIG. 8) as the comparison coordinate (comparison coordinate calculation step S23), and this can be calculated using the following formula (14).
Center X coordinate of non-reference detection frame=X coordinate of non-reference detection frame+detection frame width of non-reference detection frame×0.5 Equation (14)

The priority detection frame extraction unit C24 extracts the priority detection frame FB, which is the detection frame F that is located within the limited detection area E based on the comparison coordinates calculated by the comparison coordinate calculation unit C23. That is, the detection area limiting unit C2 in this embodiment has a priority detection frame extraction unit C24 that extracts the detection frame F whose comparison coordinates calculated by the comparison coordinate calculation unit C23 are located within the limited detection area E as the priority detection frame FB from among the non-reference detection frames F. In the example shown in FIG. 8, for the four non-reference detection frames F on the left side of the screen V among the seven non-reference detection frames F, the center X coordinate FSb, which is the comparison coordinate, is less than the minimum X coordinate value of the limited detection area E, and for the three non-reference detection frames F in the center of the screen V, the center X coordinate FSb, which is the comparison coordinate, is equal to or greater than the minimum X coordinate value of the limited detection area E and equal to or less than the maximum X coordinate value. Therefore, in the example shown in FIG. 8, the three non-reference detection frames F in the center of the screen V are extracted as the priority detection frame FB (priority detection frame extraction step S24). It can be determined from the figure that the priority detection frames FB that have comparison coordinates within the limited detection area E are all detection frames F that were used to detect the person W1, who is the priority detection object.

The above is the processing content (detection area limiting step S2) in the detection area limiting unit C2. By performing such processing, it is possible to eliminate the detection frame F relating to detection objects other than the priority detection object so that it does not affect the calculation processing by the detection coordinate calculation unit C3.

The detection coordinate calculation unit C3 includes a detection target specific coordinate setting unit C31 that uses the reference detection frame FA and the priority detection frame FB to set detection target specific coordinates FL that can identify the position of the person W, the priority detection target in the detection target image V, from each of these detection frames FA and FB (see FIG. 2). The detection coordinate calculation unit C3 does not use a non-reference detection frame F that has comparison coordinates outside the limited detection area E. Therefore, the detection target specific coordinate setting unit C31 does not set the detection target specific coordinates FL based on a non-reference detection frame F that has comparison coordinates outside the limited detection area E.

The position of each detection frame F in the detection target image V can be found as a coordinate value by a predetermined calculation as described above. In this embodiment, a rectangular detection frame F is applied in the detection unit C1, and as shown in FIG. 3, the detection target specific coordinate setting unit C31 is configured to set the position of the bottom side F1 of the detection frame F as the detection target specific coordinate FL. That is, the detection target specific coordinate setting unit C31 of this embodiment calculates the position of the feet of the person W1, who is the priority detection target in the detection target image V, as the detection target specific coordinate FL.

The reference detection frame FA and the priority detection frame FB, which have reference coordinates and comparison coordinates within the limited detection area E, include not only the detection frame F that is in close contact with the priority detection object W1, but also the detection frame F that is not in close contact with the priority detection object W1, as described above. In such a case, when the detection result detected by the detection frame F that is not in close contact with the priority detection object W1 in the detection coordinate calculation unit C3 is treated as equivalent to the detection result detected by the detection frame F that is in close contact with the priority detection object W1 like the former, and the arithmetic average is performed, and the position of the priority detection object W1 in the detection target image V is calculated as the detection coordinate, the position of the priority detection object W1 according to the calculation result becomes an inaccurate position far from the actual position of the priority detection object W1. For example, as can be seen from FIG. 9, there is a variation in the position of the detection object specific coordinate FL (the position of the priority detection object W1 in the detection target image V, which is the lower side F1 of the detection frame F in this embodiment) calculated and set by the detection object specific coordinate setting unit C31. When calculating the position of the priority detection object W1 in the detection target image V as detection coordinates (representative coordinates) from such multiple position candidates F1 (coordinates of the lower side F1 of the detection frame F), if the arithmetic average of the multiple position candidates F1 is performed and the position of the priority detection object W1 in the detection target image V is calculated as the detection coordinates, the position FL' of the priority detection object W1 based on the calculation result is affected by the coordinates of the large-sized detection frame F as shown in the figure, and there is a problem that it is calculated to be closer (closer to the camera I) than the coordinate FL (actual distance) that is originally to be calculated and shown by the dashed line in the figure. Conversely, there are cases where it is detected with a detection frame F that is smaller than the actual detection object, in which case it is calculated to be deeper (farther from the camera I) than the actual distance. As described above, when adopting a process of searching the detection target image V with the detection frame F to detect multiple detection target candidates, the calculation result is not stable if it is an arithmetic average.

To resolve such problems, in this embodiment, a predetermined weighting process is performed by the weighting process unit C32 in the detection coordinate calculation unit C3, which calculates the position of the priority detection object W1 as a detection coordinate. The predetermined weighting process by the weighting process unit C32 is a process that is performed following the process of setting the detection object specific coordinate FL by the detection object specific coordinate setting unit C31.

As shown in FIG. 2, the weighting processing unit C32 of this embodiment has a large area weighting processing unit C33 that performs a predetermined weighting for each large area obtained by dividing the entire detection target image into a plurality of areas according to the count number (density) of the detection target object identification coordinates FL (the lower side F1 of the detection frame F) that can identify the position of the priority detection target object W1 in the detection frame F, and a small area weighting processing unit C34 that performs a predetermined weighting for each small area obtained by further dividing each large area into a plurality of areas according to the count number (density) of the detection target object identification coordinates FL (the lower side F1 of the detection frame F). The processing contents of the detection coordinate calculation unit C3 that uses such large area weighting processing unit C33 and small area weighting processing unit C34 will be described below.

Figure 10(a) shows an example of horizontal lines representing the detection target identification coordinates FL (the position of the bottom side F1 of the detection frame F, the coordinate position of the feet of the person W1) calculated and set by the detection target coordinate setting unit C31 for a detection target image with a width of 720 pixels and a height of 480 pixels. In Figure 10(a), five horizontal lines are drawn, assuming that five detection target identification coordinates FL have been detected by dividing the detection target image V into multiple large regions, which are regions that are equally divided in the height direction into 10 regions. Also, Figure 10(b) is a histogram showing how many detection target identification coordinates FL belong to each large region (the count number of detection target identification coordinates FL). From this figure, it can be seen that one detection target identification coordinate FL belongs to large region 1, and four detection target identification coordinates FL belong to large region 3.

Figure 11(a) is a diagram showing the state in which large region 3, which had the largest number of counts of the detection object identification coordinates FL among the multiple large regions shown in Figure 10(a), has been further divided into eight small regions (48 pixels high) in the height direction. Figure 11(b) is a histogram showing how many detection object identification coordinates FL belong to each small region (the number of counts of the detection object identification coordinates FL). From this figure, it can be seen that three detection object identification coordinates FL belong to small region 3 and one detection object identification coordinate FL belongs to small region 4. Note that the number of large regions and the number of small regions are not limited to 10 and 8, respectively, and may be changed to any appropriate value.

The weighting processing unit C32 of this embodiment uses this information to perform weighting processing to take a weighted average, increasing the weight near the coordinates where the detection object identification coordinates FL are densely concentrated, and decreasing the weight when the detection object identification coordinates FL are far from the densely concentrated coordinates. Specifically, weighting processing is performed to multiply the count number of the detection object identification coordinates FL that can be grasped from the histogram by some power (for example, cube). The power of the weight is determined as a trade-off with the computation cost, and the larger the multiplier, the stronger the feature.

The calculation and setting process of the detection target object specific coordinate FL by the detection target object specific coordinate setting unit C31, and the detection coordinate calculation process by the detection coordinate calculation unit C3 involving weighting processing by the weighting processing unit C32 (detection coordinate calculation step S3; see FIG. 5) are performed by calculation processing using the formula shown in FIG. 12.

First, the representative coordinates of each large area are calculated using equation (1) shown in the figure. Specifically, each large area is divided into small areas, and the number of detection object identification coordinates FL (foot coordinates) belonging to each small area is multiplied by a certain power to determine the weight. The arithmetic mean of the detection object identification coordinates FL (foot coordinates) belonging to each small area is determined as the representative coordinates of the small area, and a weighted average is calculated from these. The calculation results are the representative coordinates of each large area. In this way, small area weighting processing is performed by the small area weighting processing unit C34 when calculating the representative coordinates of each large area.

Next, the detection coordinates (detection target object specific coordinates FL) of the current detection target image (current detection target image) are calculated using equation (2) shown in FIG. 12. Specifically, the number of feet coordinates belonging to each large area is multiplied by a certain power, and this is used as the weight. The representative coordinates of each large area calculated using equation (1) are used to calculate a weighted average. The calculation result is the detection target object specific coordinates FL of the current detection target image (real-time detection coordinates). In this way, large area weighting processing is performed by the large area weighting processing unit C33 when calculating the detection target object specific coordinates FL of the current detection target image.

Here, from the relationship between the mounting position and mounting angle of the imaging device I and the coordinates on the detection target image V, it is possible to calculate the distance from the imaging device I to the priority detection target W1 based on the detection target specific coordinates FL (foot coordinates) on the detection target image V. In this embodiment, a table is prepared in advance that associates the distance from the imaging device I to the detection target W with each coordinate on the detection target image V, and the system is set up so that the distance from the imaging device I to the detection target W is instantly calculated from the detection target specific coordinates FL on the detection target image V by referring to the table.

Next, the detection coordinates are calculated using the formula (3) shown in FIG. 12. Specifically, the detection coordinates of the current detection target image calculated by the formula (2) and the past detection coordinates (detection coordinates for multiple times including the most recent past) are used to calculate the detection coordinates by taking a moving average. However, if there is a person near the work machine H, a quick response is required, so the weight of the moving average is changed according to the detection coordinates of the current detection target image (see formula (3) in FIG. 12). In the example of formula (3), the dangerous distance from the imaging device I is set to 2 m, but this distance may be changed as appropriate. Furthermore, in the example of formula (3), the weight values of the current image and the nth previous image are each clearly indicated, but the weight values are not limited to this. In the example of formula (3), the number of moving averages is set to 5 (a formula using one real-time detection coordinate and five detection coordinates for four times including the most recent past), but the number of moving averages is not limited to this. In this way, in this embodiment, when calculating the detected coordinates, a method is adopted in which, depending on the distance (detected coordinates) calculated using the weighted average, a process is selected in which no weighting is applied to the moving average if the detected coordinates are greater than the danger distance, and a process in which the weighting is gradually decreased as the moving average processing progresses if the detected coordinates are equal to or less than the danger distance, and when a quick warning is required, responsiveness is prioritized over accuracy.

In this embodiment, the distance measuring unit C4 of the controller C measures (specifies) the distance from the work machine H (specifically, the imaging device I) to the priority detection object W1 by referring to the above table based on the calculation result (detection coordinates) by the detection coordinate calculation unit C3 (distance measuring step S4; see FIG. 5). The distance measuring unit C4 of this embodiment outputs the measurement result (distance to the priority detection object W1) as an image to the monitor D. In this embodiment, as an example of outputting the measurement result to the monitor D, as shown in FIG. 3, a process is adopted in which the measurement result (distance to the priority detection object W1) is displayed by a horizontal line FL that is longer in the width direction than any of the detection frames F. In particular, by displaying the horizontal line FL on the monitor D in a color different from the detection frame F, the operator can intuitively grasp the distance to the priority detection object W1. The distance measuring unit C4 may be provided with a function of outputting sound or light via an appropriate speaker or light emitter to notify or warn that a person is in a position close to the work machine H.

In this way, according to the work machine surroundings detection object position detection system X of this embodiment, when calculating the detection coordinates, the above-mentioned weighting process is performed in which the weight is increased for areas with a larger number of detection object specific coordinates FL (foot coordinates), so that the calculation result in which the position of the priority detection object W1 in the detection target image V is calculated as the detection coordinates reflects the processing content weighted by the density of the detection frame F. As a result, even if there is variation in the position candidates of the priority detection object W1 around the work machine in image recognition using the detection frame F, the variation in the detection coordinates that identify the position of the priority detection object W1 in the detection target image V is reduced, and the detection coordinates for the priority detection object W1 in the detection target image V can be calculated with high accuracy.

In particular, according to the work machine surroundings detection object position detection system X of this embodiment, even if two or more detection objects W1, W2 are detected using the detection frame F, one of the detection objects is set as the priority detection object W1, and the detection frame F related to the priority detection object W1 is extracted, so that the detection frame F related to the detection object other than the priority detection object W1 is not treated as valid data when calculating the position of the priority detection object W1 as a detection coordinate, and the position of the priority detection object W1 captured by the imaging device I can be detected with high accuracy. Specifically, according to the work machine surroundings detection object position detection system X of this embodiment, the reference coordinate FSa serving as the reference of the limited detection area E is calculated using the reference detection frame FA serving as the reference among the multiple detection frames F in the image V, and the limited detection area E is calculated based on the reference coordinate FSa, so that the limited detection area E can be set in the detection object image V by a relatively simple calculation process. In the work machine surroundings detection object position detection system X according to this embodiment, a comparison coordinate FSb is calculated for each of the non-reference detection frames F among the multiple detection frames F in the image V, and the detection frame F having the comparison coordinate FSb within the limited detection area E among the non-reference detection frames F is extracted as the priority detection frame FB. When calculating the position of the priority detection object W1, only the reference detection frame FA and the extracted priority detection frame FB are used, so that the position of the priority detection object W1 can be detected without being affected by the coordinate values of the non-reference detection frames F for detection objects other than the priority detection object W1.

In addition, according to the present embodiment, the detection system X for detecting the position of a detection target around a work machine is configured to calculate the final detection coordinates by taking a moving average of the real-time detection coordinates and the detection coordinates for multiple times including the most recent past. Therefore, compared to a mode in which the real-time detection coordinates are the final detection coordinates, the final detection coordinates can be calculated with high accuracy taking into account the movement over time for the same priority detection target W1 in the detection target image V. In particular, by adopting a configuration in which the weight of the moving average is changed (the weight is dynamically switched) according to the real-time detection coordinates, when a person who is the nearest priority detection target W1 is far from the work machine H, the weight of the moving average is set relatively large, so that the final detection coordinates can be detected with high accuracy. On the other hand, when the real-time detection coordinates are in a position close to the work machine H within a predetermined range, a dangerous situation in which a person is in a position close to the work machine H within a predetermined range can be immediately output as a calculation result, and detection processing that prioritizes responsiveness can be performed.

Although the embodiments of the present invention have been described above, the present invention is not limited to the configurations of the above embodiments. For example, in the above-mentioned embodiments, a hydraulic excavator is exemplified as a work machine, but the peripheral detection target position detection system of the present invention can also be applied to work machines other than hydraulic excavators (construction machines, agricultural and forestry work machines, and other machines that exhibit special performance for specific applications, or various vehicles other than ships and airplanes).

In the above embodiment, when multiple detection objects are detected in the detection target image by the detection process using the detection frame, the detection object closest to the work machine is set as the priority detection object and only the position of the priority detection object is detected. However, it is also possible to adopt a configuration in which a detection object other than the detection object closest to the work machine is set as the priority detection object and only the position of the priority detection object is detected, or a configuration in which a priority is determined for multiple detection objects and the above-mentioned detection area limiting process and detection coordinate calculation process are performed for each detection object (for example, a first priority detection object, a second priority detection object, a third priority detection object, etc.). In this case, first, the detection object closest to the work machine is set as the first priority detection object, and the priority detection frame extraction unit C24 of the detection area specification unit C2 extracts the priority detection frame FB for the first priority detection object. Next, the priority detection frame FB for the first priority detection object is removed from the detection frame F, and the detection object next closest to the first priority detection object from the work machine is set as the second priority detection object, and the priority detection frame FB for the second priority detection object is similarly extracted. By repeating this process according to the number of detection objects, it is possible to extract a priority detection frame for each detection object. Then, it is also possible to configure the system to measure (identify) the distance from the work machine (specifically, the imaging device) for all or any of the detection objects based on the processing result (calculation result) by the detection coordinate calculation unit.

In addition, when multiple detection objects are detected in an image, it is possible to perform a grouping process based on a specific coordinate (e.g., X coordinate) of each detection frame, and calculate the coordinates of each detection object based on the processing results.

Coordinates other than the central X-axis coordinate (for example, the central coordinate, X coordinate, Y coordinate, central Y coordinate, etc. of the reference frame or non-reference frame) may be used as the reference coordinate of the reference frame or the comparison coordinate of the non-reference frame. In addition, when calculating the limited detection area based on the reference coordinate of the reference frame, the value of the coefficient used in the calculation formula using the reference coordinate (in the above embodiment, "0.25" in (Equation 12) and "0.75" in (Equation 13)) may be changed as appropriate. This allows the range of the limited detection area to be selected and changed as appropriate.

The processing contents shown in Figures 4, 6 to 8 (processing contents of the detection unit for the detection target image, processing contents of the reference coordinate calculation unit, processing contents of the limited detection area calculation unit, and processing contents of the comparison coordinate calculation unit) are not actually displayed on the display unit D of the work machine, but may be set to be displayed on the display unit D.

In addition, in the above embodiment, an example was given of an aspect having both a large area weighting processing unit and a small area weighting processing unit, but it is also possible to adopt an aspect having only one of the weighting processing units (for example, an aspect in which the representative coordinates of the large area are calculated by arithmetic averaging, although this reduces accuracy), or to adopt an aspect in which the small area is further divided into a plurality of areas and weighting is performed for each area according to the count number of the detection target object specific coordinates.

Multiple imaging devices may be attached to one work machine, and the detection area limiting process may be performed individually for each image captured by each imaging device, or the detection coordinate calculation process may be performed with weighting. Alternatively, the current images captured by each imaging device that are taken at the same time may be combined to perform the detection area limiting process, or the detection coordinate calculation process may be performed with weighting. Multiple monitors that the operator can view may also be set.

The number of divisions in the large area and the number of divisions in the small areas in the above-mentioned embodiments can be changed as appropriate.

Furthermore, the weighting in the weighting processing unit may be weighting different from the weighted average based on the count number of the detection target object specific coordinates.

In the present invention, when multiple detection objects are detected in a detection target image using a detection frame, the position of each detection object can be detected using a detection coordinate calculation process involving the above-mentioned weighting process.

In the above embodiment, the detection target identification coordinates are coordinates corresponding to the bottom side of a rectangular detection frame, but the detection target identification coordinates may be coordinates corresponding to a side or corner of the detection frame other than the bottom side.

Feature quantities other than HOG (Histograms of Oriented Gradients), such as CoHOG, LBP, edgelet, Haar, and Joint-Haar, can be used as features used to detect objects in the target image, and methods other than SVM (Support Vector Machine), such as boosting, random trees, k-NN, Perceptron, Passive Aggressive, AROW, and Confidence weighted, can be used as learning methods used for detection training.

In addition to a monitor, a head-up display (HUD) or a head-mounted display (HMD) may also be used as the display unit.

In addition, the detection target in the present invention is not limited to a person. Even if the detection target is a living thing other than a human (such as an animal), or a small construction machine or a work support robot (for example, a disaster support robot, a ground drone, etc.), the position of the detection target can be detected by the work machine surrounding detection target position detection system and work machine surrounding detection target position detection program according to the present invention.

The specific configuration of each part is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the present invention.

C1...detection unit C2...detection area limiting unit C21...reference coordinate calculation unit C22...limited detection area calculation unit C23...comparison coordinate calculation unit C24...priority detection frame extraction unit C3...detection coordinate calculation unit C31...weighting processing unit C32...large area weighting processing unit C33...small area weighting processing unit E...limited detection area FA...reference detection frame FB...priority detection frame FSa...reference coordinate FSb...comparison coordinate H...work machine I...imaging device X...work machine surroundings detection target position detection system

Claims (5)

A work machine surrounding detection object position detection system capable of detecting a position of a detection object around a work machine, comprising:
a detection unit that detects position candidates of a detection target object in an image captured by an imaging device attached to the work machine by using detection frames of a plurality of sizes for the image;
a detection area limiting unit that limits a detection area for a priority detection object that is to be preferentially detected based on a detection result by the detection unit to within a predetermined area in the image;
a detection coordinate calculation unit that calculates a position of the priority detection object as a detection coordinate based on a detection result by the detection unit within the limited detection area limited by the detection area limiting unit,
The detection area limiting unit is
a reference coordinate calculation unit that calculates reference coordinates that serve as a reference for the limited detection area for a reference detection frame that serves as a reference among the plurality of detection frames in the image based on a detection result by the detection unit;
a limited detection area calculation unit that calculates the limited detection area based on the reference coordinates;
a comparison coordinate calculation unit that calculates comparison coordinates for each of the non-reference detection frames, which are detection frames other than the reference detection frame, among the plurality of detection frames in the video based on a detection result by the detection unit;
a priority detection frame detection unit that extracts, from the non-reference detection frames, a detection frame whose comparison coordinates calculated by the comparison coordinate calculation unit are within the limited detection area, as a priority detection frame. The detection coordinate calculation unit calculates the position of the priority detection object in the detection object image as detection coordinates based on the processing results of weighting according to the count number of detection object identification coordinates that can identify the position of the priority detection object from the reference detection frame and the priority detection frame having the comparison coordinates within the limited detection area for each of the regions into which the detection object image in the video is divided. The detection object position detection system around the work machine according to claim 1. The system for detecting the position of an object to be detected around a work machine according to claim 1 or 2, wherein the reference coordinate is the center X coordinate of the reference detection frame, and the comparison coordinate is the center X coordinate of the non-reference detection frame. A detection object position detection system around a work machine according to any one of claims 1 to 3, configured to measure the distance from the imaging device to the priority detection object based on the calculated detection coordinates. A software program for causing a computer to function as a work machine surroundings detection object position detection system for detecting the position of a detection object around a work machine,
a detection step of detecting position candidates of a detection target object in an image captured by an imaging device attached to the work machine using detection frames of a plurality of sizes for the image;
a detection area limiting step of limiting a detection area for a priority detection object to be preferentially detected based on a detection result by the detection step to within a predetermined area in the image;
a detection coordinate calculation step of calculating a position of a priority detection object in an image as a detection coordinate based on a detection result by the detection step within a limited detection area limited by the detection area limiting step,
The detection area limiting step includes:
a reference coordinate calculation step of calculating reference coordinates serving as a reference for the limited detection area for a reference detection frame serving as a reference among the plurality of detection frames in the image based on a detection result by the detection step ;
a limited detection area step of calculating the limited detection area based on the reference coordinates;
a comparison coordinate calculation step of calculating comparison coordinates for each of the non-reference detection frames, which are detection frames other than the reference detection frame, among the plurality of detection frames in the image based on a detection result in the detection step;
and a priority detection frame detection step of extracting, from the non-reference detection frames, a detection frame whose comparison coordinates calculated in the comparison coordinate calculation step are within the limited detection area, as a priority detection frame.

Images