JP7602741B2 - Systems for Excavators - Google Patents

Description

This disclosure relates to a system for an excavator.

Conventionally, there is known an excavator that recognizes the position of the rear corners of the loading platform of a dump truck in order to prevent contact between the excavation attachment of the excavator and the dump truck (see Patent Document 1).

JP 2012-21290 A

However, the above-mentioned excavator only locally recognizes the position of the rear corners of the dump truck bed, and does not recognize the size of the dump truck bed. Therefore, it cannot properly assist the loading operation.

Therefore, it is desirable to provide a system for excavators that can better assist in loading operations.

A system for a shovel according to one embodiment of the present invention is a system for a shovel having a lower running body, an upper rotating body rotatably mounted on the lower running body, and an attachment attached to the upper rotating body, and acquires point cloud data related to a dump truck based on the output of an object detection device, randomly extracts one point of interest from the point cloud data related to the dump truck, performs principal component analysis on local point cloud data around the point of interest, and detects a normal to the point of interest and a plane perpendicular to the normal and including the point of interest as planes that constitute the bed of the dump truck, thereby estimating at least one of the position, orientation, and size of the dump truck from the point cloud data related to the dump truck that belongs to the plane that constitutes the bed of the dump truck.

The above-mentioned excavator system can better assist with loading operations.

FIG. 1 is a diagram illustrating an example of the configuration of a construction support system. FIG. FIG. 2 is a diagram illustrating a configuration example of a drive system mounted on a shovel. 13 is a flowchart illustrating an example of an estimation process. FIG. 11 is a diagram illustrating an example of data used in the estimation process. FIG. 2 is a top view of a shovel and a dump truck at a construction site. 13 is a flowchart showing another example of the estimation process.

Below, the embodiment of the present invention will be described with reference to the drawings. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and the description will be omitted.

Figure 1 is a diagram showing an example of the configuration of a construction support system SYS, which is an example of a system for a shovel according to one embodiment of the present invention. The construction support system SYS is configured to recognize the position, size, orientation, etc. of a dump truck 200 as a transport vehicle, and thus to support loading operations by an excavator (shovel 100) as a construction machine. Figure 2 is a top view of the shovel 100. Figure 3 is a diagram showing an example of the configuration of a drive system mounted on the shovel 100. The drive system is a system for driving the shovel 100.

First, an overview of the excavator 100 will be described with reference to Figures 1 and 2. The excavator 100 includes a lower traveling body 1, a slewing mechanism 2, an upper rotating body 3, a boom 4, an arm 5, a bucket 6, and a cabin 10. The upper rotating body 3 is mounted on the lower traveling body 1 so as to be rotatable via the slewing mechanism 2. The boom 4, the arm 5, and the bucket 6 form an excavation attachment, which is an example of an attachment. The attachment may be a lifting magnet attachment, a grapple attachment, or the like.

As shown in FIG. 2, the lower traveling body 1 has crawlers 1C including a left crawler 1CL and a right crawler 1CR. The crawlers 1C are driven by a traveling hydraulic motor 2M. The traveling hydraulic motor 2M includes a left traveling hydraulic motor 2ML that drives the left crawler 1CL, and a right traveling hydraulic motor 2MR that drives the right crawler 1CR.

The upper rotating body 3 is driven by a swing hydraulic motor 2A. The swing hydraulic motor 2A can change the orientation of the upper rotating body 3 relative to the lower traveling body 1. The upper rotating body 3 may be electrically driven by a swing electric motor. The swing hydraulic motor 2A and the swing electric motor may be used in combination.

The boom 4 is rotatably attached to the center of the front of the upper rotating body 3. An arm 5 is rotatably attached to the tip of the boom 4, and a bucket 6 is rotatably attached to the tip of the arm 5 as an end attachment. The boom 4 is driven by a boom cylinder 7. The arm 5 is driven by an arm cylinder 8. The bucket 6 is driven by a bucket cylinder 9.

Depending on the work content, etc., other end attachments may be attached to the tip of the arm 5 instead of the bucket 6, which is an example of an end attachment. Other end attachments include, for example, a lifting magnet or a grapple.

The cabin 10 is a cab in which the operator sits. The cabin 10 is mounted on the front left side of the upper rotating body 3. If the excavator 100 is an unmanned excavator, the cabin 10 may be omitted.

Next, the functions of the excavator 100 will be described. As shown in FIG. 3, the drive system of the excavator 100 includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. As shown in FIG. 2, the hydraulic drive system of the excavator 100 includes hydraulic actuators such as a traveling hydraulic motor 2M, a swing hydraulic motor 2A, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9.

The engine 11 is a power source for the hydraulic drive system, and is mounted, for example, at the rear of the upper rotating body 3. Specifically, the engine 11 rotates at a preset target speed under direct or indirect control by the shovel controller 30 (described later), and drives the main pump 14 and the pilot pump 15. The engine 11 is, for example, a diesel engine that uses diesel as fuel.

A dial 75 is provided inside the cabin 10 of the excavator 100. The dial 75 is a dial for adjusting the engine speed, and is used, for example, to switch the engine speed in stages. In this embodiment, the dial 75 is provided so that the engine speed can be switched between four stages: SP mode, H mode, A mode, and IDLE mode. The dial 75 is configured to send data indicating the setting state of the engine speed to the excavator controller 30. Note that FIG. 3 shows a state in which the H mode is selected by the dial 75.

SP mode is the rotation speed mode selected when prioritizing the amount of work, and uses the highest engine rotation speed. H mode is the rotation speed mode selected when prioritizing both the amount of work and fuel economy, and uses the second highest engine rotation speed. A mode is the rotation speed mode selected when prioritizing fuel economy while operating the excavator with low noise, and uses the third highest engine rotation speed. IDLE mode is the rotation speed mode selected when idling the engine, and uses the lowest engine rotation speed. The engine 11 is controlled to a constant engine rotation speed that corresponds to the rotation speed mode set by the dial 75.

The main pump 14 is configured to discharge hydraulic oil. The main pump 14 is mounted on the rear of the upper rotating body 3, similar to the engine 11, and supplies hydraulic oil to the control valve unit 17 through a hydraulic oil line. The main pump 14 is driven by the engine 11. In this embodiment, the main pump 14 is a swash plate type variable displacement hydraulic pump, and the discharge flow rate (displacement volume per rotation) is controlled under the control of the excavator controller 30.

The regulator 13 is configured to control the discharge volume of the main pump 14. In this embodiment, the regulator 13 adjusts the angle (tilt angle) of the swash plate of the main pump 14 in response to a control command from the shovel controller 30.

The oil temperature sensor 14c, which is provided in the pipeline between the main pump 14 and a tank storing the hydraulic oil to be sucked in by the main pump 14, is configured to send data representing the temperature of the hydraulic oil flowing through the pipeline to the excavator controller 30.

The control valve unit 17 is a hydraulic control device including a plurality of control valves. The control valve unit 17 is mounted, for example, in the center of the upper rotating body 3, and is configured to control the hydraulic drive system in response to the operation of the operating device 26 by the operator. The control valve unit 17 is connected to the main pump 14 via a hydraulic oil line, and selectively supplies hydraulic oil supplied from the main pump 14 to one or more of the plurality of hydraulic actuators (travel hydraulic motor 2M, swing hydraulic motor 2A, boom cylinder 7, arm cylinder 8, and bucket cylinder 9) in response to the operating state of the operating device 26. Specifically, the control valve unit 17 includes a plurality of control valves (spool valves) that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each of the plurality of hydraulic actuators.

The operation system of the shovel 100 includes a pilot pump 15 and an operation device 26. The operation system of the shovel 100 also includes a shuttle valve 32 as a component related to the machine control function (described later) of the shovel controller 30.

The pilot pump 15 is mounted, for example, at the rear of the upper rotating body 3 and is configured to supply hydraulic oil to equipment such as the operating device 26 via a pilot line. In this embodiment, the pilot pump 15 is a fixed displacement hydraulic pump and is driven by the engine 11.

The operating device 26 is provided near the cockpit in the cabin 10. The operating device 26 is an operation input means used to operate the operated parts (lower traveling body 1, upper rotating body 3, boom 4, arm 5, bucket 6, etc.). That is, the operating device 26 is an operation input means used to operate the hydraulic actuators (left traveling hydraulic motor 2ML, right traveling hydraulic motor 2MR, swing hydraulic motor 2A, boom cylinder 7, arm cylinder 8, bucket cylinder 9, etc.) that drive the operated parts. The operating device 26 includes an operating lever and a traveling lever. Each of the operating devices 26 is connected to a corresponding control valve in the control valve unit 17 via a shuttle valve 32 provided in the secondary pilot line. As a result, a pilot pressure according to the operating state of the corresponding operating device 26 is input to each control valve. Therefore, each control valve operates according to the operating state of the corresponding operating device 26, and the hydraulic actuator is driven according to the operation of the corresponding control valve. The operating lever includes a left operating lever and a right operating lever. The left operating lever is used, for example, to operate the swing hydraulic motor 2A and the arm cylinder 8, and the right operating lever is used, for example, to operate the boom cylinder 7 and the bucket cylinder 9. The travel levers include a left travel lever and a right travel lever. The left travel lever is used, for example, to operate the left travel hydraulic motor 2ML, and the right travel lever is used, for example, to operate the right travel hydraulic motor 2MR. The left travel lever may be configured to be linked to the left travel pedal. The same applies to the right travel lever.

The operation device 26 is provided with a switch NS. In this embodiment, the switch NS is a push button switch. The operator can operate the operation device with his/her hand while pressing the switch NS with his/her finger. The switch NS is used, for example, when executing a machine control function.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs (flows out) the hydraulic oil having a higher pressure from the outlet port of the hydraulic oil input (flowing in) to each of the two inlet ports. One of the two inlet ports of the shuttle valve 32 is connected to the operating device 26, and the other is connected to the proportional valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve in the control valve unit 17 through a pilot line. Therefore, the shuttle valve 32 can apply the higher of the pilot pressure generated by the operating device 26 and the pilot pressure generated by the proportional valve 31 to the pilot port of the corresponding control valve. In other words, the shovel controller 30 can control the corresponding control valve to operate the corresponding operated part regardless of the operation of the operating device 26 by the operator by applying the pilot pressure generated by the proportional valve 31, which is higher than the pilot pressure generated by the operating device 26, to the pilot port of the corresponding control valve. In other words, the shovel controller 30 can operate the shovel 100 autonomously.

The operating device 26 may be an electric type that outputs an electric signal, instead of a hydraulic pilot type that generates a pilot pressure. In this case, the electric signal from the operating device 26 is input to the shovel controller 30. The shovel controller 30 can operate the hydraulic actuator according to the operation content for the operating device 26 by controlling the control valve in the control valve unit 17 according to the input electric signal. The control valve in the control valve unit 17 may be an electromagnetic solenoid type spool valve that is driven by a command from the shovel controller 30. Alternatively, a solenoid valve that operates according to an electric signal from the shovel controller 30 may be disposed between the pilot pump 15 and each pilot port of the control valve in the control valve unit 17. In this case, when manual operation is performed using the electric operating device 26, the shovel controller 30 can operate the corresponding solenoid valve by an electric signal corresponding to the operation amount (e.g., the lever operation amount). Specifically, the excavator controller 30 can operate the corresponding control valve in accordance with the amount of operation of the operating device 26 by increasing or decreasing the pilot pressure generated by the hydraulic oil discharged by the pilot pump 15 using a solenoid valve.

If the shovel 100 is an unmanned shovel, the operating device 26 may be omitted. Alternatively, if the shovel 100 is a remote-controlled shovel, the operating device 26 may be installed outside the shovel 100 (cabin 10).

In this embodiment, the control system of the shovel 100 includes a discharge pressure sensor 28, an operating pressure sensor 29, a shovel controller 30, a proportional valve 31, a display device 40, an input device 42, a sound output device 43, a memory device 47, a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a machine body inclination sensor S4, a turning state sensor S5, a spatial recognition device S6, a positioning device P1, and a communication device T1.

The shovel controller 30, which is an example of a control device, is provided, for example, in the cabin 10 and is configured to control the drive of the shovel 100. The shovel controller 30 may be realized by hardware, software, or a combination thereof. In this embodiment, the shovel controller 30 is configured by a computer including a CPU, a volatile storage device, a non-volatile storage device, and various input/output interfaces. The shovel controller 30 realizes various functions by executing various programs stored in the non-volatile storage device.

For example, the shovel controller 30 sets a target rotation speed of the engine 11 based on a work mode or the like that is set in advance by a predetermined operation by an operator or the like, and controls the engine 11 to rotate at a constant speed. Also, for example, the shovel controller 30 outputs a control command to the regulator 13 as necessary to change the discharge amount of the main pump 14. Also, for example, the shovel controller 30 may execute a machine guidance function that guides (guides) the manual operation of the shovel 100 by the operator through the operation device 26. Also, the shovel controller 30 may execute a machine control function that automatically assists the manual operation of the shovel 100 by the operator through the operation device 26, or operates the shovel 100 independently of manual operation. Alternatively, the machine control function may be a function for operating an unmanned shovel.

In addition, some of the functions of the shovel controller 30 may be realized by other controllers (control devices). That is, the above-mentioned functions may be realized in a manner in which they are distributed and processed by multiple controllers. For example, the machine guidance function and the machine control function may be realized by a dedicated controller (control device).

In this embodiment, the shovel controller 30 is configured to estimate at least one of the position, orientation, and size of the dump truck 200 (hereinafter referred to as "position, etc."). The estimation of the position, etc. of the dump truck 200 by the shovel controller 30 will be described in detail later.

The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. The discharge pressure sensor 28 transmits a detection signal related to the discharge pressure to the excavator controller 30.

The operating pressure sensor 29, which is an example of an operating state detection device, is configured to detect the operating state of the operating device 26. In this embodiment, the operating pressure sensor 29 detects the pilot pressure generated by the operating device 26. The operating pressure sensor 29 transmits a detection signal related to the operating pressure to the excavator controller 30.

The operation state detection device may be an encoder, potentiometer, tilt sensor, or the like capable of detecting the tilt direction and tilt angle of the operating lever.

The proportional valve 31 is provided in a pilot line connecting the pilot pump 15 and the shuttle valve 32, and is configured so that its flow area (the cross-sectional area through which hydraulic oil can flow) can be changed. The proportional valve 31 operates in response to a control command input from the shovel controller 30. With this configuration, even when the operating lever is not being operated, for example, the shovel controller 30 can supply hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17 via the proportional valve 31 and shuttle valve 32, thereby operating the hydraulic actuator.

The display device 40 is configured to be able to display various images. In this embodiment, the display device 40 is provided in a position that is easily visible to an operator seated in the cockpit inside the cabin 10, and displays various images under the control of the shovel controller 30. The display device 40 may be connected to the shovel controller 30 via an on-board communication network such as a CAN, or may be connected to the shovel controller 30 via a one-to-one dedicated line.

In this embodiment, the display device 40 operates by receiving power from the storage battery 70. The storage battery 70 is charged with power generated by the alternator 11a of the engine 11. The power of the storage battery 70 is also supplied to the shovel controller 30, the shovel's electrical equipment 72, and other components other than the display device 40. In addition, the starter 11b of the engine 11 is driven by power from the storage battery 70 to start the engine 11.

The engine control device 74 is configured to control the engine 11. The engine control device 74 transmits various data indicating the state of the engine 11 (e.g., data indicating the cooling water temperature detected by the water temperature sensor 11c) to the shovel controller 30. The shovel controller 30 stores this data in an internal temporary storage unit (memory) and can transmit it to the display device 40 as appropriate.

The input device 42 is a device for inputting information to the control device. In this embodiment, the input device 42 is provided within reach of an operator seated in the cockpit inside the cabin 10, accepts input from the operator, and outputs a signal corresponding to the input to the shovel controller 30. The input device 42 may be a touch panel, a knob switch provided at the end of an operating lever, a button switch, a lever, a toggle, or a rotary dial provided around the display device 40, or the like. The input device 42 transmits a signal related to the input to the shovel controller 30.

The sound output device 43 is configured to output sound. In this embodiment, the sound output device 43 is provided in the cabin 10 and is connected to the shovel controller 30. The sound output device 43 is configured to output sound under the control of the shovel controller 30. Specifically, the sound output device 43 is a speaker or a buzzer. The sound output device 43 audibly outputs various information in response to an output command from the shovel controller 30.

The storage device 47 is configured to store information. In this embodiment, the storage device 47 is provided in the cabin 10 and is configured to store various information under the control of the shovel controller 30. Specifically, the storage device 47 is a non-volatile storage medium such as a semiconductor memory. The storage device 47 may store information output by various devices during operation of the shovel 100, or may store information acquired via various devices before the operation of the shovel 100 is started. The storage device 47 may store data related to a target construction surface acquired via the communication device T1 or the like, or set via the input device 42 or the like. The target construction surface may be set by the operator of the shovel 100, or may be set by a construction manager or the like.

The boom angle sensor S1 is attached to the boom 4 and is configured to detect the rotation angle of the boom 4 relative to the upper rotating body 3 (hereinafter referred to as the "boom angle"). The boom angle sensor S1 transmits a detection signal related to the boom angle to the excavator controller 30.

The arm angle sensor S2 is attached to the arm 5 and is configured to detect the rotation angle of the arm 5 relative to the boom 4 (hereinafter referred to as the "arm angle"). The arm angle sensor S2 transmits a detection signal related to the arm angle to the shovel controller 30.

The bucket angle sensor S3 is attached to the bucket 6 and is configured to detect the rotation angle of the bucket 6 relative to the arm 5 (hereinafter referred to as the "bucket angle"). The bucket angle sensor S3 transmits a detection signal related to the bucket angle to the shovel controller 30.

The boom angle sensor S1 is composed of at least one of, for example, a rotary encoder, an acceleration sensor, a gyro sensor, or an inertial measurement device. The boom angle sensor S1 may be a potentiometer using a variable resistor, a cylinder stroke sensor that detects the stroke amount of a hydraulic cylinder (boom cylinder 7) that corresponds to the boom angle, etc. The same applies to the arm angle sensor S2 and the bucket angle sensor S3.

The machine body tilt sensor S4 is configured to detect the inclination state of the machine body relative to a predetermined virtual plane. The predetermined virtual plane is, for example, a virtual horizontal plane. The machine body is, for example, the upper rotating body 3. The machine body tilt sensor S4 is, for example, attached to the upper rotating body 3, and detects a roll angle, which is the inclination angle around the front-to-rear axis of the upper rotating body 3, and a pitch angle, which is the inclination angle around the left-to-right axis of the upper rotating body 3. The machine body tilt sensor S4 may be, for example, at least one of an acceleration sensor, a gyro sensor, an inertial measurement unit, and the like. The machine body tilt sensor S4 then transmits a detection signal regarding the inclination state to the shovel controller 30.

The rotation state sensor S5 is configured to detect the rotation state of the upper rotating body 3. The rotation state sensor S5 detects, for example, the rotation angular velocity of the upper rotating body 3. The rotation state sensor S5 may detect the rotation angle of the upper rotating body 3. Specifically, the rotation state sensor S5 is a gyro sensor, a resolver, a rotary encoder, or the like. The rotation state sensor S5 transmits a detection signal related to the rotation angle or the rotation angular velocity of the upper rotating body 3 to the excavator controller 30.

The spatial recognition device S6 is configured to recognize the state of the space to be monitored. The state of the space includes the presence or absence of an object in the space. In this embodiment, the spatial recognition device S6 is configured to recognize the state of the space around the shovel 100. Specifically, the spatial recognition device S6 includes a sensor S6F that recognizes the state of the space in front of the shovel 100, a sensor S6L that recognizes the state of the space to the left of the shovel 100, a sensor S6R that recognizes the state of the space to the right of the shovel 100, and a sensor S6B that recognizes the state of the space behind the shovel 100.

Sensor S6F is attached to the front end of the upper surface of the cabin 10, sensor S6L is attached to the left end of the upper surface of the upper rotating body 3, sensor S6R is attached to the right end of the upper surface of the upper rotating body 3, and sensor S6B is attached to the rear end of the upper surface of the upper rotating body 3.

In this embodiment, the spatial recognition device S6 is a LIDAR. Specifically, the sensor S6F is composed of a combination of multiple LIDARs. The sensor S6F may be composed of one LIDAR. The same applies to the sensors S6B, S6L, and S6R. Furthermore, each of the sensors S6B, S6F, S6L, and S6R may include a monocular camera. Furthermore, the spatial recognition device S6 may be an RGB-D sensor, a stereo camera, a distance image camera, an infrared sensor, an ultrasonic sensor, a millimeter wave radar, or a laser radar, or may be a monocular camera having an imaging element such as a CCD or CMOS and a wide-angle lens attached thereto. The spatial recognition device S6 transmits the acquired information to the shovel controller 30. The spatial recognition device S6 may transmit the acquired information to the display device 40.

The spatial recognition device S6 may function as an object detection device. In this case, the spatial recognition device S6 may detect an object present around the shovel 100. The object to be detected may be, for example, a person, an animal, a vehicle, a construction machine, a building, or a hole. The spatial recognition device S6 may also calculate the distance between the spatial recognition device S6 or the shovel 100 and the detected object. In this case, the spatial recognition device S6 may also function as an object detection device. When the spatial recognition device S6 is an ultrasonic sensor, a millimeter wave radar, a laser radar, a LIDAR, an RGB-D sensor, an infrared sensor, or the like, the spatial recognition device S6 may transmit a number of signals (laser light, etc.) toward the surroundings and receive the reflected signals to detect the distance between the spatial recognition device S6 and the object and the direction in which the object is present.

In this embodiment, the spatial recognition device S6 is directly connected to the shovel controller 30. However, the spatial recognition device S6 may be connected to the display device 40 and indirectly connected to the shovel controller 30 via the display device 40.

The positioning device P1 is configured to measure the position of the upper rotating body 3. The positioning device P1 may be configured to detect the orientation of the upper rotating body 3. In this embodiment, the positioning device P1 is a GNSS compass, and derives the position and orientation of the upper rotating body 3. The positioning device P1 then transmits information regarding the position and orientation of the upper rotating body 3 to the excavator controller 30. The orientation of the upper rotating body 3 may be detected based on the output of a direction sensor attached to the upper rotating body 3.

The communication device T1 is configured to communicate with an external device through a communication network including at least one of a mobile communication network, a satellite communication network, a short-range wireless communication network, and an Internet network. The communication device T1 is, for example, a mobile communication module compatible with a mobile communication standard such as LTE, 4G, or 5G, a short-range wireless communication module compatible with a short-range wireless communication standard such as Wi-Fi or Bluetooth (registered trademark), or a satellite communication module for connecting to a satellite communication network.

Next, the details of the construction support system SYS will be described with reference to FIG. 1. The construction support system SYS mainly includes a shovel 100, a dump truck 200, a management device 300, a support device 400, and a monitoring device 500. The shovel 100 (shovel controller 30), dump truck 200 (dump controller 210), management device 300 (control device 310), support device 400 (control device 410), and monitoring device 500 (control device 510) are configured to be able to communicate via a communication network 900.

The dump truck 200 is an example of a transport vehicle. In this embodiment, the dump truck 200 is a vehicle for transporting a load loaded by the excavator 100, and mainly includes a dump controller 210, a display device 220, a communication device 230, and a positioning device 240. The load is, for example, soil, gravel, crushed stone, asphalt, concrete, etc. (hereinafter referred to as "soil, etc.").

The dump controller 210 is configured by a computer including a CPU, a volatile storage device, a non-volatile storage device, and various input/output interfaces. The dump controller 210 realizes various functions by, for example, executing various programs stored in the non-volatile storage device. The dump controller 210 is configured to control the display device 220, the communication device 230, and the positioning device 240.

In this embodiment, the dump truck controller 210 has an image generation unit. The image generation unit is configured to generate an image including an image related to a target stop position (described later). Specifically, the image generation unit receives information related to the target stop position calculated by the shovel controller 30 via the communication device 230. The image generation unit then generates an image that enables the driver of the dump truck 200 to stop the dump truck 200 at the target stop position, and causes the display device 220 to display the image.

The display device 220 is configured to display an image. In this embodiment, the display device 220 is provided in a position that is easily visible to the driver seated in the driver's seat, and displays various images under the control of the dump controller 210. The display device 220 may be connected to the dump controller 210 via an in-vehicle communication network such as a CAN, or may be connected to the dump controller 210 via a one-to-one dedicated line.

The communication device 230 is configured to communicate with an external device through a communication network including at least one of a mobile communication network, a satellite communication network, a short-range wireless communication network, and an Internet network. The communication device 230 is, for example, a mobile communication module compatible with a mobile communication standard such as LTE, 4G, or 5G, a short-range wireless communication module compatible with a short-range wireless communication standard such as Wi-Fi or Bluetooth, or a satellite communication module for connecting to a satellite communication network.

The positioning device 240 is configured to measure the position of the dump truck 200. The positioning device 240 may be configured to detect the orientation of the dump truck 200. In this embodiment, the positioning device 240 is a GNSS compass, and derives the position and orientation of the dump truck 200. The positioning device 240 then transmits information regarding the position and orientation of the dump truck 200 to the dump controller 210. The orientation of the dump truck 200 may be detected based on the output of a direction sensor attached to the dump truck 200.

The management device 300 is a terminal device installed outside the construction site. In this embodiment, the management device 300 is a stationary computer installed in a management center or the like outside the construction site, and mainly includes a control device 310, a display device 320, and a communication device 330. However, the management device 300 may also be a notebook PC, a tablet PC, or a mobile terminal such as a smartphone.

The control device 310 is configured by a computer including a CPU, a volatile storage device, a non-volatile storage device, and various input/output interfaces. The control device 310 realizes various functions by, for example, executing various programs stored in the non-volatile storage device. The control device 310 is configured to control the display device 320 and the communication device 330.

The display device 320 is configured to display images. In this embodiment, the display device 320 displays various images under the control of the control device 310.

The communication device 330 is configured to communicate with an external device through a communication network including at least one of a mobile communication network, a satellite communication network, a short-range wireless communication network, and an Internet network. In this embodiment, the communication device 330 is a mobile communication module compatible with a mobile communication standard such as LTE, 4G, or 5G, a short-range wireless communication module compatible with a short-range wireless communication standard such as Wi-Fi or Bluetooth, or a satellite communication module for connecting to a satellite communication network.

The support device 400 is a mobile terminal, such as a notebook PC, tablet PC, or smartphone carried by a worker at the construction site. In this embodiment, the support device 400 is a smartphone carried by the driver of the dump truck 200, and mainly includes a control device 410, a display device 420, a communication device 430, and a positioning device 440. The control device 410, the display device 420, and the communication device 430 are similar to the control device 310, the display device 320, and the communication device 330 in the management device 300.

In this embodiment, the control device 410 has an image generation unit, similar to the dump controller 210. The image generation unit is configured to generate an image including an image related to a target stop position (described later). Specifically, the image generation unit receives information related to the target stop position calculated by the shovel controller 30 via the communication device 430. The image generation unit then generates an image that enables the driver of the dump truck 200 to stop the dump truck 200 at the target stop position, and causes the display device 420 to display the image.

The positioning device 440 is configured to measure the position of the support device 400. The positioning device 440 may be configured to detect the orientation of the dump truck 200 when the support device 400 is attached to a cradle provided near the driver's seat of the dump truck 200. In this case, the positioning device 440 derives the orientation of the dump truck 200 based on the output of a direction sensor attached to the support device 400, for example. The positioning device 440 then transmits information regarding the position and orientation of the dump truck 200 to the control device 410.

The monitoring device 500 is configured to monitor a predetermined area. In this embodiment, the monitoring device 500 is mounted on an aircraft such as a multicopter (drone) or an airship, and is configured to be able to monitor the construction site from the sky. Specifically, the monitoring device 500 mainly has a control device 510, a communication device 530, a positioning device 540, and a spatial recognition device 550. The control device 510 and the communication device 530 are similar to the control device 310 and the communication device 330 in the management device 300. The monitoring device 500 may be attached to the top of a tall structure such as a steel tower installed at the construction site, and monitor the area around the structure from the sky.

The positioning device 540 is configured to measure the position of the monitoring device 500. The positioning device 540 may be configured to detect the orientation of the monitoring device 500. In this embodiment, the positioning device 540 is a GNSS compass and derives the position and orientation of the monitoring device 500. The positioning device 540 then transmits information regarding the position and orientation of the monitoring device 500 to the control device 510. The orientation of the monitoring device 500 may be detected based on the output of a direction sensor attached to the monitoring device 500.

The spatial recognition device 550 is configured to recognize the state of the space to be monitored, similar to the spatial recognition device S6. In this embodiment, the spatial recognition device 550 is configured to capture an image of the construction site from the sky. Specifically, the spatial recognition device 550 is, for example, a monocular camera, a stereo camera, a distance imaging camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, a laser radar, a LIDAR, an RGB-D sensor, or an infrared sensor. In this embodiment, the spatial recognition device 550 is a monocular camera attached to a multicopter. The spatial recognition device 550 may function as an object detection device, similar to the spatial recognition device S6. In this case, the spatial recognition device 550 may detect an object present in the construction site.

Next, the process in which the shovel controller 30 estimates the position, etc., of the dump truck 200 (hereinafter referred to as the "estimation process") will be described with reference to Figures 4 and 5. Figure 4 is a flowchart showing an example of the estimation process. The shovel controller 30 repeatedly executes this estimation process at a predetermined control period. Figure 5 shows an example of data used in the estimation process.

First, the shovel controller 30 extracts a point cloud related to the dump truck 200 (step ST1). In this embodiment, the shovel controller 30 extracts point cloud data related to the dump truck 200 from the point cloud data output by the LIDAR serving as the sensor S6F. The sensor S6F may be composed of multiple adjacent LIDARs.

Point cloud data is an example of three-dimensional data. In this embodiment, the point cloud data is represented as a distance image. A distance image is an image in which the distance from a specific point is used as the value of each element of a two-dimensional array.

For example, the shovel controller 30 extracts point cloud data related to the dump truck 200 by removing point cloud data related to objects other than the dump truck 200, such as the ground or materials.

For example, the shovel controller 30 may extract the difference between a distance image acquired at a first time point and a distance image acquired at a second time point after the first time point as point cloud data related to the dump truck 200. That is, point cloud data related to an object that moved during the time between the first time point and the second time point may be extracted as point cloud data related to the dump truck 200. Alternatively, the shovel controller 30 may extract point cloud data whose height from the ground is equal to or greater than a predetermined height as point cloud data related to the dump truck 200. Alternatively, the shovel controller 30 may extract point cloud data related to the dump truck 200 using any other method.

Figure 5 (A) shows an image 200G of the dump truck 200 captured by a monocular camera attached to a LIDAR as the spatial recognition device S6. Figure 5 (B) shows point cloud data PG1 related to the dump truck 200 extracted by the shovel controller 30. The point cloud data PG1 shown in Figure 5 (B) corresponds to the image 200G of the dump truck 200 shown in Figure 5 (A). Note that the image 200G of the dump truck 200 captured by the monocular camera is not essential data for performing the estimation process. Therefore, capturing the image of the dump truck 200 by the monocular camera may be omitted, or the monocular camera itself may be omitted.

Then, the shovel controller 30 performs a process of removing the bottom surface of the loading platform and creates a histogram of normals (step ST2). In this embodiment, the shovel controller 30 uses principal component analysis to determine a normal a at a point of interest from the point of interest and its neighboring points in the point cloud. That is, the shovel controller 30 determines a normal a at each of the points of interest that constitute the point cloud relating to the five surfaces that constitute the loading platform of the dump truck 200. The five surfaces are the bottom surface, the front surface (front panel), the rear surface (rear gate), the left side surface (left side gate), and the right side surface (right side gate). The shovel controller 30 then determines the angle θ formed between the normal a and the vertically upward normal b using the following formula (1).

ダンプトラック２００の荷台の形状は、略直方体であるため、ショベルコントローラ３０は、ダンプトラック２００の荷台の前面、後面、左側面、右側面、及び底面の５つの面に対して角度θを算出できる。ダンプトラック２００の荷台の底面は、荷台の側面とは異なり、ダンプトラック２００の傾き、空間認識装置Ｓ６とダンプトラック２００との間の距離、又は、空間認識装置Ｓ６から見たダンプトラック２００の向き等の変化に応じて死角となる部分の範囲が変化し易い。そのため、本実施形態では、ショベルコントローラ３０は、荷台の底面に関する点群データを除去した上で、残りの点群データに基づいてダンプトラック２００の位置等を推定するように構成されている。

Since the shape of the bed of the dump truck 200 is a substantially rectangular parallelepiped, the shovel controller 30 can calculate the angle θ for five surfaces, namely, the front surface, rear surface, left side surface, right side surface, and bottom surface of the bed of the dump truck 200. Unlike the side surfaces of the bed, the bottom surface of the bed of the dump truck 200 is prone to change in the range of the blind spot depending on the inclination of the dump truck 200, the distance between the spatial recognition device S6 and the dump truck 200, or the direction of the dump truck 200 as seen from the spatial recognition device S6. Therefore, in this embodiment, the shovel controller 30 is configured to remove the point cloud data related to the bottom surface of the bed and then estimate the position of the dump truck 200 based on the remaining point cloud data.

When the dump truck 200 is located on a horizontal plane, the normal to the bottom surface of the loading platform faces vertically upward. If the angle θ calculated by equation (1) is equal to or less than a predetermined value, the shovel controller 30 can determine that the normal a has approximately the same direction as the vertically upward normal b, and can determine that the point of interest at that time is one of the point clouds related to the bottom surface of the loading platform. Therefore, by calculating the angle θ, the shovel controller 30 can remove the point cloud data related to the bottom surface of the loading platform from the point cloud data related to the dump truck 200. In this way, the shovel controller 30 can extract the surfaces for which the angle θ calculated by equation (1) is greater than a predetermined value as the remaining four surfaces of the loading platform.

Figure 5 (C) shows point cloud data PG2 obtained by removing point cloud data relating to the bottom surface of the bed from point cloud data PG1 relating to dump truck 200. Specifically, Figure 5 (C) shows point cloud data PG2 relating to the four surfaces that make up the bed of dump truck 200. The four surfaces are the front surface (front panel), rear surface (rear gate), left side surface (left side gate), and right side surface (right side gate).

Then, based on the point cloud data relating to the remaining four surfaces of the loading platform, the shovel controller 30 creates a two-dimensional histogram in which the horizontal axis represents the z-component, which is the component in the depth direction of the normal (the front-to-rear direction of the shovel 100), and the vertical axis represents the angle of the normal with respect to the depth direction. In this embodiment, the point cloud data at this time excludes the point cloud data relating to the bottom surface of the loading platform.

Point cloud data contains errors due to noise, and normals calculated from the point cloud data also contain errors due to noise. Therefore, in this embodiment, the shovel controller 30 assumes that the variation in error in the point cloud follows a Gaussian distribution, and assumes that a two-dimensional histogram created from the normals follows a mixed Gaussian distribution. The shovel controller 30 then estimates the normals of each surface by clustering the point clouds using an Expectation-Maximization (EM) algorithm. For example, the shovel controller 30 sets the normal corresponding to the class containing the largest number of point clouds as a representative normal, and performs clustering on the point clouds belonging to that class using the three-dimensional distance to nearby points as a threshold value.

Then, the shovel controller 30 detects the surface (step ST3). In this embodiment, the shovel controller 30 detects the point cloud related to the dump truck 200 after the point cloud related to the bottom surface of the loading platform has been removed as a surface. Specifically, the shovel controller 30 detects at least one of the four surfaces that constitute the loading platform of the dump truck 200.

More specifically, the excavator controller 30 detects, as a surface, a point cloud in which the number of points constituting the clustered point cloud as described above is equal to or greater than a predetermined value and which belongs to a class in which the three-dimensional distance is the smallest.

Furthermore, to determine whether the detected face corresponds to a short side (the front or rear face of the loading platform) or a long side (the left or right side of the loading platform) of the rectangle, the shovel controller 30 extracts a normal that forms an angle of 90 degrees with respect to the normal of the detected face. The shovel controller 30 then detects other faces by clustering using the three-dimensional distance to nearby points, as described above. The shovel controller 30 then determines whether each face corresponds to a short side or a long side of the rectangle based on the length of each detected face.

Then, the shovel controller 30 estimates the normal (step ST4). In this embodiment, the shovel controller 30 performs normal estimation using RANSAC (Random Sample Consensus). Specifically, the shovel controller 30 performs normal estimation using RANSAC for the faces corresponding to the short sides (or long sides) obtained from the representative normal, distinguishing the faces from the faces corresponding to the long sides (or short sides).

The point cloud relating to the detected surface typically contains a lot of noise, and the direction of the derived normal also contains errors. Therefore, in this embodiment, the shovel controller 30 is configured to reduce errors caused by the influence of noise by performing normal estimation using RANSAC for the representative surface.

Then, the shovel controller 30 estimates the position, etc. of the dump truck 200 (step ST5). In this embodiment, the shovel controller 30 detects the faces that make up the bed from the direction of the normal and the center of gravity of the faces obtained as described above, and estimates the position, etc. of the dump truck 200 based on the position and orientation of the detected faces. Because the shape of the bed of the dump truck 200 is approximately a rectangular parallelepiped, the shovel controller 30 can estimate the position and orientation of the dump truck 200 by determining the shape of the faces that make up the approximately rectangular parallelepiped.

5(D) shows a rectangle RS representing the right side of the bed of the dump truck 200, which is one of the surfaces detected by the shovel controller 30. Specifically, the rectangle RS corresponds to the inner surface of the right side gate. In this case, the shovel controller 30 can estimate the size (tonnage or vehicle type, etc.) of the dump truck 200 from the three-dimensional distance of at least one of the four sides of the rectangle RS, for example. In this case, the shovel controller 30 may derive the size of the dump truck 200 by referring to a table that systematically stores the relationship between the three-dimensional distance of at least one of the four sides of the rectangle RS and the size of the dump truck 200. For example, the shovel controller 30 may refer to the table using the three-dimensional distance of the long side of the rectangle RS representing the right side of the bed of the dump truck 200 (corresponding to the length of the long side of the right side gate) as a search key, and identify the vehicle type of the dump truck 200 equipped with a right side gate of such a length from the vehicle type information input in advance. The shovel controller 30 may then derive the size of the dump truck 200 (e.g., maximum load capacity, vehicle length, vehicle width, or vehicle height) stored in association with the identified vehicle type of the dump truck 200. The shovel controller 30 can also estimate the orientation of the dump truck 200 from, for example, the orientation of the normal to the rectangle RS. The shovel controller 30 can also estimate the position of the dump truck 200 from, for example, the output of the positioning device P1 and the three-dimensional distance between at least one of the four vertices of the rectangle RS and the spatial recognition device S6.

The shovel controller 30 may be configured to estimate at least one of the position, orientation, and size of the dump truck 200 based on information on the two or more detected surfaces. For example, the shovel controller 30 may derive a rectangle representing the rear surface of the loading platform of the dump truck 200 in addition to the rectangle RS representing the right side of the loading platform of the dump truck 200. The rectangle representing the rear surface of the loading platform of the dump truck 200 corresponds to, for example, the outer surface of the rear gate. In this case, the shovel controller 30 may finally estimate the position, etc. of the dump truck 200 based on, for example, the position, etc. of the dump truck 200 provisionally estimated from the rectangle RS representing the right side of the loading platform and the position, etc. of the dump truck 200 provisionally estimated from the rectangle representing the rear surface of the loading platform. Alternatively, the shovel controller 30 may estimate a part of the position, etc. of the dump truck 200 (for example, only the orientation) from the rectangle RS representing the right side of the loading platform, and estimate the remainder of the position, etc. of the dump truck 200 (for example, the position and size) from the rectangle representing the rear surface of the loading platform.

This estimation process allows the shovel controller 30 to accurately estimate the position of the dump truck 200, etc. Therefore, when executing, for example, a machine control function, the shovel controller 30 can operate the hydraulic actuator of the shovel 100 appropriately and autonomously. Specifically, the shovel controller 30 can increase the reliability of the loading operation by the shovel 100 by appropriately and autonomously operating the hydraulic actuator mounted on the shovel 100. In the above embodiment, the shovel controller 30 detects the dump truck 200 from the rear, so the bottom surface of the loading platform is removed and the right side surface is extracted, but when the dump truck 200 is detected from above, extracting the bottom surface can increase the estimation accuracy of the position of the dump truck 200, etc. In this case, the shovel controller 30 can remove the front, rear, left side, and right side surfaces, and determine the position and orientation of the dump truck 200 based on the extracted bottom surface. In this way, the surface to be removed and the surface to be extracted are changed depending on the orientation in which the dump truck 200 is detected.

Next, the effect of the estimation process will be described with reference to FIG. 6. FIG. 6 shows an example of a work site where the shovel 100 is loading earth and sand into the dump truck 200. Specifically, FIG. 6 is a top view of the shovel 100 and the dump truck 200 at the construction site. In FIG. 6, the shovel 100 drawn with a solid line represents the state of the shovel 100 when the excavation operation is completed, and the shovel 100 drawn with a dashed line represents the state of the shovel 100 before the soil discharge operation is started. In this embodiment, the soil discharge operation is a bucket opening operation or a combined operation including the bucket opening operation, and does not include a swing operation. Typically, the soil discharge operation is a combined operation including the bucket opening operation, the arm opening operation, and the boom lowering operation. In addition, the bucket 6A drawn with a solid line represents the state of the bucket 6 when the excavation operation is completed, and the bucket 6B drawn with a dashed line represents the state of the bucket 6 before the soil discharge operation is started. In addition, the thick dashed line in FIG. 6 represents the trajectory drawn by a predetermined point on the back of the bucket 6.

In loading operations, the operator of the shovel 100 normally uses the operating device 26 to perform a boom-raising and swinging operation after completing an excavation operation. In this embodiment, the operator performs a combined operation including a right swing operation. Specifically, the operator performs a combined operation including a boom-raising operation and a right swing operation until the posture of the shovel 100 becomes the posture shown by the dashed line, that is, until a predetermined point on the back surface of the bucket 6 reaches point P2. The combined operation may also include an opening and closing operation of the bucket 6. This is to move the bucket 6 onto the bed of the dump truck 200 while preventing the bed and the bucket 6 from coming into contact with each other. By performing this series of operations, the operator can load soil and sand into the bed of the dump truck 200.

The rectangle CB represented by the thick dotted line represents the position of the bed of the dump truck 200 estimated by the shovel controller 30 through the estimation process. The two-dot chain line represents the orientation of the dump truck 200 estimated by the shovel controller 30. Specifically, the two-dot chain line represents the front-rear axis L1 of the dump truck 200.

As shown in FIG. 6, by estimating the position of the dump truck 200, the shovel controller 30 can recognize that, for example, the rotation axis SA of the shovel 100 and the longitudinal axis L1 of the dump truck 200 are in a positional relationship where they do not intersect. In other words, the shovel controller 30 can recognize that no matter how the rotation position of the upper rotating body 3 is changed, the longitudinal axis of the upper rotating body 3 and the longitudinal axis L1 of the dump truck 200 cannot be made parallel.

When it is recognized that such a positional relationship exists, the excavator controller 30 can recognize that, for example, unless the upper rotating body 3 is rotated appropriately during the soil discharge operation, the soil and sand taken into the bucket 6 cannot be discharged evenly onto the bed of the dump truck 200.

When starting the soil discharge operation, if the front-rear axis of the upper rotating body 3 and the front-rear axis L1 of the dump truck 200 are parallel, the excavator 100 can evenly spread the soil and sand taken into the bucket 6 on the bed of the dump truck 200 simply by performing the soil discharge operation. This is because the bucket 6 can be moved along the front-rear axis L1 of the dump truck 200. However, if the front-rear axis of the upper rotating body 3 and the front-rear axis L1 of the dump truck 200 are not parallel, the excavator 100 cannot evenly spread the soil and sand taken into the bucket 6 on the bed of the dump truck 200 simply by performing the soil discharge operation. This is because the bucket 6 cannot be moved along the front-rear axis L1 of the dump truck 200 simply by operating the excavation attachment AT. Therefore, when the shovel controller 30 recognizes that the rotation axis SA of the shovel 100 and the front-rear axis L1 of the dump truck 200 are in a positional relationship where they do not intersect, the shovel controller 30 can autonomously rotate the upper rotating body 3 during the soil discharge operation so that the soil and sand taken into the bucket 6 is evenly spread on the bed of the dump truck 200. In other words, the operator of the shovel 100 can evenly spread the soil and sand taken into the bucket 6 on the bed of the dump truck 200 without performing a swing operation during the soil discharge operation.

Alternatively, when the shovel controller 30 recognizes that the rotation axis SA of the shovel 100 and the front-rear axis L1 of the dump truck 200 are in a positional relationship in which they do not intersect, the shovel controller 30 may drive the traveling hydraulic motor 2M to operate the lower traveling body 1. Specifically, the shovel controller 30 may move the shovel 100 so that the rotation axis SA of the shovel 100 and the front-rear axis L1 of the dump truck 200 intersect.

Alternatively, the shovel controller 30 may inform the operator of the shovel 100 that the rotation axis SA of the shovel 100 and the front-rear axis L1 of the dump truck 200 are in a positional relationship in which they do not intersect. This is to prompt the operator to move the shovel 100 so that the rotation axis SA of the shovel 100 and the front-rear axis L1 of the dump truck 200 intersect.

Alternatively, the shovel controller 30 may inform the driver of the dump truck 200 that the rotation axis SA of the shovel 100 and the front-rear axis L1 of the dump truck 200 are in a positional relationship where they do not intersect. This is to prompt the driver to move the dump truck 200 so that the rotation axis SA of the shovel 100 and the front-rear axis L1 of the dump truck 200 intersect. In this case, the shovel controller 30 may derive a target stop position and present information about the target stop position to the driver of the dump truck 200. The target stop position is a position where the dump truck 200 should be stopped, and is represented by, for example, a rectangle corresponding to the occupation area of the dump truck 200 when the dump truck 200 is viewed from above. The target stop position is typically set so that the front-rear axis L1 of the dump truck 200 stopped at the target stop position intersects with the rotation axis SA of the shovel 100.

In this way, being able to accurately estimate the position of the dump truck 200 etc. can improve the efficiency of loading operations by the shovel 100. This is because the shovel controller 30 can act on at least one of the operator of the shovel 100 and the driver of the dump truck 200 so that loading operations are performed with the rotation axis SA of the shovel 100 and the front-rear axis L1 of the dump truck 200 intersecting.

Next, another example of the estimation process will be described with reference to FIG. 7. FIG. 7 is a flowchart showing another example of the estimation process. The shovel controller 30 repeatedly executes this estimation process at a predetermined control period. The estimation process shown in FIG. 7 differs from the estimation process shown in FIG. 4 in that it includes step ST0, but is the same as the estimation process shown in FIG. 4 in that it includes steps ST1 to ST5. Therefore, in the following, a description of the common parts will be omitted and the different parts will be described in detail.

In the estimation process shown in FIG. 7, the shovel controller 30 determines whether the dump truck 200 is located within a predetermined working range (step ST0). In this embodiment, the shovel controller 30 executes this determination task before extracting a point cloud related to the dump truck 200 using the output of the spatial recognition device S6. That is, the shovel controller 30 does not extract a point cloud related to the dump truck 200 until it is determined that the dump truck 200 is located within the predetermined working range. This is to prevent unnecessary calculations from being performed, such as accurately estimating the position of the dump truck 200, even when the dump truck 200 is located away from the shovel 100 (a position unsuitable for loading operations).

Specifically, the shovel controller 30 determines whether the dump truck 200 is located within a predetermined work range based on an image captured by a monocular camera attached to the spatial recognition device S6. The shovel controller 30 may determine the presence or absence of the dump truck 200 based on the output of a device other than the monocular camera, such as an ultrasonic sensor or a millimeter wave radar. That is, the shovel controller 30 may determine the presence or absence of the dump truck 200 based on the output of any device, including a LIDAR, as long as the determination task of step ST0 can be executed with a lower computation cost than the computation cost for executing the tasks after step ST1. At this stage, the shovel controller 30 only needs to be able to determine whether the dump truck 200 exists around the shovel 100, and even if the dump truck 200 exists around the shovel 100, it is sufficient to be able to roughly recognize the position of the dump truck 200. For example, the shovel controller 30 may determine whether the dump truck 200 is present or recognize the position of the dump truck 200 by pattern matching, which compares a preregistered image of the dump truck 200 with an image portion included in an image captured by the monocular camera. Alternatively, the shovel controller 30 may determine whether an image of the dump truck 200 is present in an image captured by the monocular camera or recognize the position of the dump truck 200 by using deep learning. The shovel controller 30 recognizes, for example, the coordinates of the center of the loading platform of the dump truck 200 or the coordinates of the rear end of the loading platform of the dump truck 200 as the position of the dump truck 200. The coordinates are, for example, coordinates in a coordinate system centered on the position of the monocular camera.

The shovel controller 30 may also recognize the position and orientation of the dump truck 200 based on an image captured by the monocular camera. In this case, the shovel controller 30 only needs to roughly recognize the position and orientation of the dump truck 200.

The predetermined working range is, for example, a range suitable for loading operations. In other words, the dump truck 200 being located within the predetermined working range means that the dump truck 200 is located within a range suitable for loading operations. In this embodiment, the predetermined working range is set as a coordinate range in a coordinate system centered on the position of the monocular camera.

The shovel controller 30 determines whether the dump truck 200 is located within the predetermined work range based on the recognized position of the dump truck 200 and the set predetermined work range. For example, the shovel controller 30 determines that the dump truck 200 is located within the predetermined work range when the coordinates of the center of the loading platform of the dump truck 200 are included within the coordinate range corresponding to the predetermined work range.

If it is determined that the dump truck 200 is not located within the specified working range (NO in step ST0), the excavator controller 30 ends the current estimation process without executing the tasks from step ST1 onwards.

On the other hand, if it is determined that the dump truck 200 is located within the specified work range (YES in step ST0), the shovel controller 30 executes the tasks from step ST1 onwards and estimates the position of the dump truck 200, etc.

In addition, once it is determined that the dump truck 200 is located within the predetermined working range, in the next and subsequent estimation processes, the shovel controller 30 may omit the determination of step ST0 and execute the tasks of step ST1 and subsequent steps. For example, during a predetermined period after it is determined that the dump truck 200 is located within the predetermined working range, or until the estimation process has been repeated a predetermined number of times, the shovel controller 30 may omit the determination of step ST0 and execute the tasks of step ST1 and subsequent steps. This is to reduce the computational load.

This configuration allows the shovel controller 30 to reduce the computational load related to estimating the position, etc., of the dump truck 200. This is because when there is no dump truck 200 around the shovel 100, or when the dump truck 200 is located in a location that is not suitable for loading operations, the shovel controller 30 does not execute tasks subsequent to step ST1, which are tasks for estimating the position, etc., of the dump truck 200.

As described above, the construction support system SYS, which is a system for a shovel 100 having a lower traveling body 1, an upper rotating body 3 rotatably mounted on the lower traveling body 1, and an attachment attached to the upper rotating body 3 according to one embodiment of the present invention, is configured to acquire three-dimensional data regarding the dump truck 200 based on the output of an object detection device, and estimate at least one of the position, orientation, and size of the dump truck 200 based on information regarding at least one surface constituting the bed of the dump truck 200. The object detection device is typically a spatial recognition device S6 attached to the shovel 100. However, the object detection device may be a spatial recognition device installed outside the shovel 100. For example, the object detection device may be attached to another construction machine, a building, a utility pole, an aircraft, or the like.

With this configuration, the construction support system SYS can more appropriately support the loading operation by the excavator 100. This is because at least one of the position, orientation, and size of the dump truck 200 can be accurately estimated. In addition, the construction support system SYS can estimate at least one of the position, orientation, and size of the dump truck 200 without performing image recognition processing using machine learning such as deep learning, so there is no need to prepare a trained model, etc., and the system can be easily introduced.

The construction support system SYS may be configured to extract information about at least one surface constituting the bed of the dump truck 200 from the three-dimensional data related to the dump truck 200. The at least one surface constituting the bed of the dump truck 200 is typically at least one of the four side surfaces constituting the bed of the dump truck 200.

The construction support system SYS may be configured to extract three-dimensional data relating to the bed of the dump truck 200 from the three-dimensional data relating to the dump truck 200. For example, the construction support system SYS may be configured to extract three-dimensional data relating to the bed of the dump truck 200 by removing three-dimensional data relating to the soil and sand around the dump truck 200, the road surface, or the tires of the dump truck 200, from the three-dimensional data output by the spatial recognition device S6.

When extracting information about at least one surface constituting the bed of the dump truck 200, the construction support system SYS may be configured to preferentially extract information about a surface whose overall shape can be recognized. In this case, the surface whose overall shape can be recognized may be, for example, a surface whose three or more corners can be recognized. Alternatively, the surface whose overall shape can be recognized may be, for example, a surface whose two or more corners can be recognized. Typically, the surface whose overall shape can be recognized is at least one of the four side surfaces constituting the bed of the dump truck 200. When viewed from the spatial recognition device S6, the bottom surface of the bed of the dump truck 200 is largely hidden by the side surfaces constituting the bed of the dump truck 200, so in the above-mentioned embodiment, the construction support system SYS does not preferentially extract the bottom surface of the bed of the dump truck 200. However, in another embodiment, the construction support system SYS may preferentially extract the bottom surface of the bed of the dump truck 200.

With this configuration, the construction support system SYS can estimate at least one of the position, orientation, and size of the dump truck 200 based on information about the surface whose overall shape can be recognized, thereby improving the reliability of the estimation result.

The information about at least one surface constituting the bed of the dump truck 200 is preferably information about a surface other than the bottom surface of the bed of the dump truck 200. This is because, as described above, when viewed from the spatial recognition device S6, much of the bottom surface of the bed of the dump truck 200 is hidden in the shadow of the side surface constituting the bed of the dump truck 200.

The construction support system SYS may be configured to derive the size or type of the dump truck 200 based on information about the shape of at least one surface that constitutes the bed of the dump truck 200, among information about at least one surface that constitutes the bed of the dump truck 200. The information about the shape of the surface is, for example, the three-dimensional distance of the long side or short side of a rectangle.

The construction support system SYS may be configured to derive the position of the dump truck 200 based on information about the position of at least one surface that constitutes the bed of the dump truck 200, among information about at least one surface that constitutes the bed of the dump truck 200. The information about the position of the surface is, for example, the coordinates of the center of gravity of the surface, or the coordinates of a point on the contour of the surface.

The construction support system SYS may be configured to derive the orientation of the dump truck 200 based on information about the orientation of at least one surface that constitutes the bed of the dump truck 200, among information about at least one surface that constitutes the bed of the dump truck 200. The information about the orientation of the surface is, for example, the orientation of the normal to the surface, or the extension direction of the long side or short side of the surface, etc.

The construction support system SYS may be configured to determine whether or not the dump truck 200 exists before acquiring three-dimensional data related to the dump truck 200. Alternatively, the construction support system SYS may be configured to estimate at least one of the position, orientation, and type of the dump truck 200 before acquiring three-dimensional data related to the dump truck 200. Alternatively, the construction support system SYS may be configured to determine whether or not the dump truck 200 has entered a predetermined work range before acquiring three-dimensional data related to the dump truck 200. For example, the construction support system SYS may be configured to determine whether or not the dump truck 200 exists based on an image captured by a camera mounted on the shovel 100. In this case, the camera may be a monocular camera or a stereo camera. This is to prevent a task with a high computational cost for accurately estimating the position, etc. of the dump truck 200 from being executed even though the dump truck 200 is away from the shovel 100. With this configuration, the construction support system SYS can prevent the computational load from increasing unnecessarily.

The above describes the preferred embodiments of the present invention in detail. However, the present invention is not limited to the above-described embodiments. Various modifications or substitutions can be applied to the above-described embodiments without departing from the scope of the present invention. Furthermore, features described separately can be combined as long as no technical contradiction occurs.

For example, in the above embodiment, the shovel controller 30 is configured to remove point cloud data related to the bottom surface of the bed of the dump truck 200 in the estimation process. However, the shovel controller 30 may be configured to remove point cloud data related to another surface other than the bottom surface of the bed of the dump truck 200. For example, in a situation where the spatial recognition device S6 looks down on the dump truck 200 from a high position, the bottom surface of the bed of the dump truck 200 tends to have a narrower blind spot. On the other hand, for example, the rear surface of the bed of the dump truck 200 is prone to change in the blind spot depending on changes in the distance between the spatial recognition device S6 and the dump truck 200. In this case, the shovel controller 30 may be configured to remove point cloud data related to the rear surface of the bed of the dump truck 200 in the estimation process.

1: Lower traveling body 1C: Crawler 1CL: Left crawler 1CR: Right crawler 2: Swing mechanism 2A: Swing hydraulic motor 2M: Travel hydraulic motor 2ML: Left traveling hydraulic motor 2MR: Right traveling hydraulic motor 3: Upper rotating body 4: Boom 5: Arm 6: Bucket 7: Boom cylinder 8: Arm cylinder 9: Bucket cylinder 10: Cabin 11: Engine 11a: Alternator 11b: Starter 11c: Water temperature sensor 13: Regulator 14: Main pump 14c: Oil temperature sensor 15: Pilot pump 17: Control valve unit 26: Operation device 28: Discharge pressure sensor 29: Operation pressure sensor 30: Excavator controller 31: Proportional valve 32: Shuttle valve 40: Display device 42: Input device 43: Sound output device 47: Memory device 70: Storage battery 72: Electrical equipment 74: Engine control device 75: Dial 100: Shovel 200: Dump truck 210: Dump controller 220: Display device 230: Communication device 240: Positioning device 300: Management device 310: Control device 320: Display device 330: Communication device 400: Support device 410: Control device 420: Display device 430: Communication device 440: Positioning device 500: Monitoring device 510: Control device 530: Communication device 540: Positioning device 550: Spatial recognition device 900: Communication network NS: Switch P1: Positioning device S1: Boom angle sensor S2: Arm angle sensor S3: Bucket angle sensor S4: Machine body inclination sensor S5: Turning state sensor S6: Spatial recognition device S6B, S6F, S6L, S6R: Sensor SYS: Construction support system T1: Communication device

Claims (13)

A lower running body;
An upper rotating body rotatably mounted on the lower traveling body;
and an attachment attached to the upper rotating body,
acquiring point cloud data related to the dump truck based on the output of an object detection device, randomly extracting one point of interest from the point cloud data related to the dump truck, performing principal component analysis on local point cloud data around the point of interest, and detecting a normal line at the point of interest and a plane perpendicular to the normal line and including the point of interest as a plane constituting the bed of the dump truck, thereby estimating at least one of the position, orientation, and size of the dump truck from the point cloud data related to the dump truck belonging to the plane constituting the bed of the dump truck;
System for excavators. A lower running body;
An upper rotating body rotatably mounted on the lower traveling body;
and an attachment attached to the upper rotating body,
Point cloud data relating to the dump truck is acquired based on the output of an object detection device, and before detecting at least one surface constituting the bed of the dump truck and after detecting at least one other surface constituting the bed of the dump truck, point cloud data belonging to at least one other surface constituting the bed of the dump truck is removed , thereby preventing the point cloud data belonging to at least one other surface constituting the bed of the dump truck from being used in detecting at least one surface constituting the bed of the dump truck .
System for excavators. At least one other surface constituting the bed of the dump truck from which the point cloud data is removed is changed depending on the orientation of the dump truck.
A system for a shovel according to claim 2. At least one other surface among the surfaces constituting the bed of the dump truck is at least one of a bottom surface, a front surface, a rear surface, a left side surface, and a right side surface.
A system for a shovel according to claim 2. Estimating at least one of a position, a direction, and a size of the dump truck based on information about a surface, the overall shape of which can be recognized, among at least one surface constituting the bed of the dump truck;
At least one surface constituting the bed of the dump truck is rectangular,
The surface on which the overall shape can be recognized is a surface on which three or more corners can be recognized.
A system for a shovel according to any one of claims 1 to 4. The information regarding at least one surface constituting the bed of the dump truck is information regarding a surface other than the bottom surface of the bed of the dump truck.
A system for a shovel according to any one of claims 1 to 5. Deriving the size or type of the dump truck based on information about the shape of at least one surface that constitutes the bed of the dump truck, or
Deriving a position of the dump truck based on information about the position of at least one surface constituting the bed of the dump truck, or
Deriving an orientation of the dump truck based on information regarding an orientation of at least one surface constituting a bed of the dump truck;
A system for a shovel according to any one of claims 1 to 6. estimating a part of the position, orientation, and size of the dump truck based on information about at least one surface constituting the bed of the dump truck, and estimating the remaining part of the position, orientation, and size of the dump truck based on information about at least one other surface constituting the bed of the dump truck;
A system for a shovel according to any one of claims 1 to 7. At least one of the position, orientation, and size of the dump truck is estimated based on information about at least one surface constituting the bed of the dump truck, and at least one of the position, orientation, and size of the dump truck is estimated based on information about at least one other surface constituting the bed of the dump truck, estimating at least one of the position, orientation, and size of the dump truck;
A system for a shovel according to any one of claims 1 to 8. determining whether the dump truck is present or not before acquiring point cloud data relating to the dump truck;
The position of the dump truck when it is determined that the dump truck is present is farther from the shovel than the position of the dump truck when acquisition of point cloud data regarding the dump truck is started.
A system for a shovel according to any one of claims 1 to 9. Estimating at least one of a position, an orientation, and a type of the dump truck before acquiring point cloud data related to the dump truck;
A position of the dump truck when at least one of the position, orientation, and type of the dump truck is estimated before acquiring point cloud data related to the dump truck is farther away from a shovel than a position of the dump truck when acquisition of the point cloud data related to the dump truck is started.
A system for a shovel according to any one of claims 1 to 10. determining whether the dump truck has entered a predetermined range;
The position of the dump truck when the dump truck enters a predetermined range is farther from the shovel than the position of the dump truck when acquisition of point cloud data regarding the dump truck is started.
A system for a shovel according to any one of claims 1 to 11. determining whether the dump truck is present or not based on an image captured by a camera mounted on a shovel;
The position of the dump truck when it is determined that the dump truck is present based on an image captured by a camera mounted on a shovel is farther from the shovel than the position of the dump truck when acquisition of point cloud data regarding the dump truck is started.
A system for a shovel according to any one of claims 1 to 12.

Images