JP7799952B2 - Support devices, work machines, and programs - Google Patents

Description

This disclosure relates to support devices for work machines, etc.

For example, a technology has been disclosed in which teaching points are set and a trajectory for the working part of a work machine is generated based on those teaching points (see Patent Document 1).

Japanese Patent Application Laid-Open No. 2021-50576

However, for example, in Patent Document 1, it is necessary to set teaching points by actually operating a shovel and operating the work machine, which can result in a lot of time and effort being required to generate the target trajectory for the work area.

In view of the above issues, the objective is to provide technology that can more easily generate the trajectory of the working part of a work machine.

In order to achieve the above object, in one embodiment of the present disclosure,
an acquisition unit that acquires data relating to the shape of a work target in the vicinity of the work machine;
a display unit that displays an image representing the shape of the work object based on the data acquired by the acquisition unit;
a setting unit that sets a point on the shape of the work object that will be a target when the work machine is operating;
a generation unit that generates a trajectory of a working portion of the work machine based on the data acquired by the acquisition unit, the target shape of the work object, and the points set by the setting unit,
Support devices are provided.

In another embodiment of the present disclosure,
The above-mentioned support device is provided.
A work machine is provided.

In still another embodiment of the present disclosure,
Support devices include:
an acquisition step of acquiring data relating to the shape of a work object around the work machine;
a display step of displaying an image representing the shape of the work object on a display unit based on the data acquired in the acquisition step;
a setting step of setting a point on the shape of the work object that will be a target when the work machine is operating;
a generating step of generating a trajectory of a working portion of the work machine based on the data acquired in the acquiring step, the target shape of the work object, and the points set in the setting step;
Programs are offered.

The above-described embodiment makes it easier to generate the trajectory of the working part of a work machine.

FIG. 1 illustrates an example of an operation support system. FIG. 1 is a top view showing an example of a shovel. FIG. 2 is a diagram illustrating an example of a configuration for remotely controlling a shovel. FIG. 2 is a block diagram showing an example of a hardware configuration of the shovel. FIG. 2 illustrates an example of a hardware configuration of an information processing device. FIG. 2 is a functional block diagram showing an example of a functional configuration related to generation of a target trajectory of a working part of a shovel. FIG. 10 is a diagram showing an example of a screen related to generation of a target trajectory of a working part of a shovel. FIG. 10 is a diagram showing another example of a screen relating to generation of a target trajectory of a working part of a shovel. FIG. 10 is a diagram showing yet another example of a screen relating to generation of a target trajectory of a working part of a shovel. 10 is a flowchart illustrating an example of a process for generating a target trajectory of a working portion of a shovel.

The following describes the embodiment with reference to the drawings.

[Outline of the operation support system]
First, an overview of the operation support system SYS according to this embodiment will be described with reference to FIGS.

Figure 1 is a diagram showing an example of an operation support system SYS. In Figure 1, a left side view of the shovel 100 is shown. Figure 2 is a top view showing an example of the shovel 100. Figure 3 is a diagram showing an example of the configuration related to remote operation of the shovel. Hereinafter, the direction in which the attachment AT extends when viewed from above the shovel 100 (upward in Figure 2) will be defined as "front," and directions on the shovel 100 or directions seen from the shovel 100 may be described.

As shown in FIG. 1, the operation support system SYS includes an excavator 100 and an information processing device 200.

The operation support system SYS uses the information processing device 200 to work in conjunction with the excavator 100 and provide support regarding the operation of the excavator 100.

The operation support system SYS may include one or more excavators 100.

The excavator 100 is a work machine that receives operation support in the operation support system SYS.

As shown in Figures 1 and 2, the excavator 100 comprises a lower traveling body 1, an upper rotating body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10.

The undercarriage 1 uses crawlers 1C to propel the excavator 100. The crawlers 1C include a left crawler 1CL and a right crawler 1CR. The crawler 1CL is hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawler 1CL is hydraulically driven by a traveling hydraulic motor 1MR. This allows the undercarriage 1 to travel independently.

The upper rotating body 3 is rotatably mounted on the lower traveling body 1 via the rotating mechanism 2. For example, the upper rotating body 3 rotates relative to the lower traveling body 1 when the rotating mechanism 2 is hydraulically driven by a rotating hydraulic motor 2M.

The boom 4 is attached to the front center of the upper rotating body 3 so that it can be raised and lowered around a rotation axis that runs in the left-right direction. The arm 5 is attached to the tip of the boom 4 so that it can rotate around a rotation axis that runs in the left-right direction. The bucket 6 is attached to the tip of the arm 5 so that it can rotate around a rotation axis that runs in the left-right direction.

The bucket 6 is an example of an end attachment and is used, for example, for excavation work.

The bucket 6 is attached to the tip of the arm 5 in a manner that allows it to be replaced as needed depending on the type of work being performed by the excavator 100. In other words, instead of the bucket 6, a different type of bucket, such as a relatively large bucket, a slope bucket, or a dredging bucket, may be attached to the tip of the arm 5. Furthermore, an end attachment other than a bucket, such as a mixer, breaker, or crusher, may also be attached to the tip of the arm 5. Furthermore, a spare attachment, such as a quick coupling or tiltrotator, may be provided between the arm 5 and the end attachment.

The boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, arm cylinder 8, and bucket cylinder 9, respectively.

The cabin 10 is a control room where the operator sits and operates the excavator 100. The cabin 10 is mounted, for example, on the front left side of the upper rotating body 3.

For example, the excavator 100 operates driven elements such as the lower traveling structure 1 (i.e., the pair of left and right crawlers 1CL, 1CR), upper rotating structure 3, boom 4, arm 5, and bucket 6 in response to operations by an operator seated in the cabin 10.

Furthermore, instead of or in addition to being configured to be operable by an operator inside the cabin 10, the shovel 100 may be configured to be remotely operable from outside the shovel 100. When the shovel 100 is remotely operated, the interior of the cabin 10 may be unmanned. The following explanation will be given on the assumption that operator operation includes at least one of operation of the operating device 26 by the operator inside the cabin 10 and remote operation by an external operator.

For example, as shown in FIG. 3, remote operation includes a mode in which the shovel 100 is operated by operation input related to the actuator of the shovel 100 performed by the remote operation support device 300.

The remote operation support device 300 is installed, for example, in a management center that externally manages the work of the shovel 100. The remote operation support device 300 may also be a portable operation terminal, in which case the operator can remotely operate the shovel 100 while directly checking the work status of the shovel 100 from the vicinity of the shovel 100.

The shovel 100 may transmit, for example, via the communication device 60 described below, an image (hereinafter referred to as a "peripheral image") showing the surrounding area including the area in front of the shovel 100, based on an image captured by the imaging device 40 described below. The remote operation support device 300 may then display the image (peripheral image) received from the shovel 100 on a display device. Various information images (information screens) displayed on the output device 50 (display device 50A) inside the cabin 10 of the shovel 100 may also be displayed on the display device of the remote operation support device 300. This allows the operator using the remote operation support device 300 to remotely operate the shovel 100 while checking, for example, the contents of the images and information screens showing the surrounding area of the shovel 100 displayed on the display device. The excavator 100 may operate actuators and drive driven elements such as the lower traveling body 1, upper rotating body 3, boom 4, arm 5, and bucket 6 in response to remote control signals indicating the content of the remote control, which are received by the communication device 60 from the remote operation support device 300.

Remote control may also include, for example, a mode in which the shovel 100 is operated by external voice input or gesture input to the shovel 100 by a person (e.g., a worker) around the shovel 100. Specifically, the shovel 100 recognizes voices uttered by surrounding workers and gestures made by the workers through a voice input device (e.g., a microphone) or a gesture input device (e.g., an imaging device) mounted on the shovel 100. The shovel 100 may then operate actuators in accordance with the content of the recognized voices and gestures to drive driven elements such as the lower traveling body 1 (left and right crawlers 1C), upper rotating body 3, boom 4, arm 5, and bucket 6.

The work of the shovel 100 may also be remotely monitored. In this case, a remote monitoring support device having the same functions as the remote operation support device 300 may be provided. The remote monitoring support device is, for example, the information processing device 200. This allows a monitor, who is the user of the remote monitoring support device, to monitor the status of the work of the shovel 100 while checking the surrounding image displayed on the display device of the remote monitoring support device. Furthermore, for example, if the monitor determines it is necessary from a safety perspective, the monitor can intervene in the operation of the operator of the shovel 100 and cause an emergency stop by making a specified input using the input device of the remote monitoring support device.

The information processing device 200 communicates with the shovel 100 to cooperate with it and provide support for the operation of the shovel 100.

The information processing device 200 is a server or management terminal device installed, for example, in an administrative office within the work site of the shovel 100, or in a management center that manages the operating status of the shovel 100 and is located in a location different from the work site of the shovel 100. The management terminal device may be, for example, a fixed terminal device such as a desktop PC (Personal Computer), or a portable terminal device (mobile terminal) such as a tablet terminal, smartphone, or laptop PC. In the latter case, workers at the work site, supervisors who oversee the work, managers who manage the work site, etc. can move around the work site carrying the portable information processing device 200. In the latter case, the operator can, for example, bring the portable information processing device 200 into the cabin of the shovel 100.

The information processing device 200, for example, acquires data relating to the operating status from the shovel 100. This allows the information processing device 200 to grasp the operating status of the shovel 100 and monitor the presence or absence of abnormalities in the shovel 100. The information processing device 200 can also display data relating to the operating status of the shovel 100 via the display device 208 (described below) so that the user can check it.

The information processing device 200 also transmits to the shovel 100 various data, such as programs and reference data used in the processing of the controller 30, etc., to the shovel 100. This allows the shovel 100 to perform various processes related to the operation of the shovel 100 using the various data downloaded from the information processing device 200.

[Hardware configuration of the operation support system]
Next, the hardware configuration of the operation support system SYS will be described with reference to FIGS. 4 and 5 in addition to FIGS. 1 to 3.

<Excavator hardware configuration>
FIG. 4 is a block diagram showing an example of the hardware configuration of the shovel 100.

In Figure 4, the paths through which mechanical power is transmitted are indicated by double lines, the paths through which high-pressure hydraulic oil that drives the hydraulic actuator flows are indicated by solid lines, the paths through which pilot pressure is transmitted are indicated by dashed lines, and the paths through which electrical signals are transmitted are indicated by dotted lines.

The excavator 100 includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

<Hydraulic drive system>
4, the hydraulic drive system of the excavator 100 includes hydraulic actuators HA that hydraulically drive each of the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, and the attachment AT, as described above. The hydraulic drive system of the excavator 100 according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

The hydraulic actuator HA includes travel hydraulic motors 1ML, 1MR, swing hydraulic motor 2M, boom cylinder 7, arm cylinder 8, and bucket cylinder 9.

In addition, the excavator 100 may have some or all of the hydraulic actuators HA replaced with electric actuators. In other words, the excavator 100 may be a hybrid excavator or an electric excavator.

The engine 11 is the prime mover of the excavator 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, at the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by the controller 30 (described below), and drives the main pump 14 and pilot pump 15.

In addition, other prime movers (e.g., electric motors) may be installed in the excavator 100 instead of or in addition to the engine 11.

The regulator 13 controls (adjusts) the discharge rate of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilt angle") in response to a control command from the controller 30.

The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, at the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13, thereby adjusting the piston stroke length and controlling the discharge flow rate and discharge pressure.

The control valve 17 drives the hydraulic actuators HA in response to the operator's operation of the operating device 26, the details of remote operation, or operation commands corresponding to the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to each hydraulic actuator in response to the operator's operation or operation commands corresponding to the automatic operation function. Specifically, the control valve 17 includes multiple control valves (also referred to as "directional control valves") that control the flow rate and direction of hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

≪Operation system≫
As shown in FIG. 4 , the operating system of the excavator 100 includes a pilot pump 15 , an operating device 26 , a hydraulic control valve 31 , a shuttle valve 32 , and a hydraulic control valve 33 .

The pilot pump 15 supplies pilot pressure to various hydraulic equipment via a pilot line 25. The pilot pump 15 is mounted, for example, at the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 may be reduced in pressure by a specified pressure reducing valve, and the resulting relatively low-pressure hydraulic oil may be supplied to various hydraulic equipment as pilot pressure.

The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate the various driven elements. Specifically, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive each driven element, thereby enabling the operator to operate the driven elements that are driven by the hydraulic actuators HA. The operating device 26 includes pedal devices and lever devices for operating each driven element (hydraulic actuator HA).

For example, as shown in FIG. 4, the operating device 26 is a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and a pilot line 25A branching from the pilot line 25, and outputs a pilot pressure corresponding to the operation to the secondary pilot line 27A. The pilot line 27A is connected to one inlet port of the shuttle valve 32 and to the control valve 17 via the pilot line 27, which is connected to the outlet port of the shuttle valve 32. This allows pilot pressure corresponding to the operation of the various driven elements (hydraulic actuators HA) in the operating device 26 to be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by an operator or the like.

The operating device 26 may also be electrical. In this case, the pilot line 27A, shuttle valve 32, and hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electrical signal (hereinafter referred to as the "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. The controller 30 then outputs a control command corresponding to the operation signal, i.e., a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26.

In addition, the control valves (directional control valves) built into the control valve 17 that drive each hydraulic actuator HA may be of the electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be input directly to the control valve 17, i.e., the electromagnetic solenoid type control valve.

Furthermore, as described above, some or all of the hydraulic actuators HA may be replaced with electric actuators. In this case, the controller 30 may output control commands to the electric actuators or a driver that drives the electric actuators, depending on the operation content of the operating device 26 and the remote operation content specified by the remote operation signal. Furthermore, if the excavator 100 is remotely operated, the operating device 26 may be omitted.

A hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA) (e.g., the raising and lowering directions of the boom 4). In other words, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA. The hydraulic control valve 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17 and configured to change its flow area (i.e., the cross-sectional area through which hydraulic oil can flow). This allows the hydraulic control valve 31 to output a predetermined pilot pressure to the secondary pilot line 27B using hydraulic oil from the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure to the control valve 17 in response to a control signal from the controller 30 via the shuttle valve 32 between the pilot line 27B and the pilot line 27B. Therefore, the controller 30 causes the hydraulic control valve 31 to supply pilot pressure to the control valve 17 in accordance with the operation command corresponding to the automatic operation function, thereby enabling the operation of the excavator 100 using the automatic operation function.

The controller 30 may also control, for example, the hydraulic control valve 31 to realize remote operation of the excavator 100. Specifically, the controller 30 outputs a control signal corresponding to the remote operation content specified in the remote operation signal received from the remote operation assistance device 300 via the communication device 60 to the hydraulic control valve 31. As a result, the controller 30 causes the hydraulic control valve 31 to supply a pilot pressure corresponding to the remote operation content to the control valve 17, thereby realizing operation of the excavator 100 based on remote operation by the operator.

Furthermore, if the operating device 26 is electric, the controller 30 can cause the hydraulic control valve 31 to directly supply pilot pressure to the control valve 17 in accordance with the operation content (operation signal) of the operating device 26, thereby realizing operation of the excavator 100 based on the operator's operation.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic fluid having the higher pilot pressure of the two pilot pressures input to the two inlet ports to the outlet port. A shuttle valve 32 is provided for each driven element (hydraulic actuator HA) operated by the operating device 26 and for each drive direction of the driven element (hydraulic actuator HA). One of the two inlet ports of the shuttle valve 32 is connected to the secondary pilot line 27A of the operating device 26 (specifically, the lever device or pedal device included in the operating device 26), and the other is connected to the secondary pilot line 27B of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected via the pilot line 27 to the pilot port of the corresponding control valve of the control valve 17. The corresponding control valve is the control valve that drives the hydraulic actuator operated by the lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure in the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure in the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31, the controller 30 can control the corresponding control valve regardless of the operator's operation of the operating device 26. Therefore, the controller 30 can control the operation of the driven elements (undercarriage 1, upper rotating body 3, attachment AT) regardless of the operating state of the operating device 26 by the operator, thereby achieving remote operation and automatic driving functions.

The hydraulic control valve 33 is provided in the pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to change its flow path area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. This allows the controller 30 to forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by the operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to a value lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply the desired pilot pressure to the pilot port of the control valve in the control valve 17, regardless of the operation of the operating device 26. Therefore, for example, by controlling hydraulic control valve 33 in addition to hydraulic control valve 31, controller 30 can more appropriately realize the remote control function and automatic operation function of excavator 100.

<User Interface>
As shown in FIG. 4 , the user interface system of the excavator 100 includes an operation device 26 , an output device 50 , and an input device 52 .

The output device 50 outputs various information to the user of the excavator 100 (e.g., the operator of the cabin 10 or an external remote control operator) and people in the vicinity of the excavator 100 (e.g., workers or drivers of work vehicles).

For example, the output device 50 includes lighting equipment and a display device 50A (see FIG. 5) that visually output various types of information. The lighting equipment is, for example, a warning light (indicator lamp). The display device 50A is, for example, a liquid crystal display or an organic EL (electroluminescence) display. For example, as shown in FIG. 2, the lighting equipment and display device 50A may be provided inside the cabin 10 and output various types of information visually to an operator or the like inside the cabin 10. The lighting equipment and display device 50A may also be provided, for example, on the side of the upper rotating body 3 and output various types of information visually to workers or the like around the excavator 100.

Furthermore, for example, the output device 50 includes a sound output device that outputs various information auditorily. Sound output devices include, for example, buzzers and speakers. The sound output device may be provided, for example, inside or outside the cabin 10, and may output various information auditorily to an operator inside the cabin 10 or to people (workers, etc.) around the excavator 100.

Furthermore, for example, the output device 50 may include a device that outputs various information in a tactile manner, such as by vibrating the cockpit.

The input device 52 accepts various inputs from the user of the excavator 100, and signals corresponding to the accepted inputs are input into the controller 30. The input device 52 may be provided, for example, inside the cabin 10, and accept inputs from an operator or the like inside the cabin 10. The input device 52 may also be provided, for example, on the side of the upper rotating body 3, and accept inputs from workers or the like near the excavator 100.

For example, the input device 52 includes an operation input device that accepts input from a user through mechanical operation. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), etc.

Furthermore, for example, the input device 52 may include an audio input device that accepts audio input from the user. The audio input device may include, for example, a microphone.

Furthermore, for example, the input device 52 may include a gesture input device that accepts gesture input from the user. The gesture input device may include, for example, an imaging device that captures images of gestures made by the user.

Furthermore, for example, the input device 52 may include a biometric input device that accepts biometric input from the user. Biometric input includes, for example, input of biometric information such as the user's fingerprint or iris.

<Communications>
As shown in FIG. 4 , the communication system of the shovel 100 according to this embodiment includes a communication device 60 .

The communication device 60 connects to an external communication line and communicates with devices provided separately from the shovel 100. The devices provided separately from the shovel 100 may include devices external to the shovel 100, as well as portable terminal devices (mobile terminals) brought into the cabin 10 by the user of the shovel 100. The communication device 60 may include, for example, a mobile communication module conforming to standards such as 4G ( ^4th Generation) and 5G ( ^5th Generation). The communication device 60 may also include, for example, a satellite communication module. The communication device 60 may also include, for example, a Wi-Fi communication module or a Bluetooth (registered trademark) communication module. The communication device 60 may also include multiple communication devices in accordance with the communication lines to be connected.

For example, the communication device 60 communicates with external devices such as the information processing device 200 and remote operation support device 300 at the work site through a local communication line established at the work site. The local communication line is, for example, a local 5G (so-called local 5G) mobile communication line established at the work site or a local network (LAN: Local Area Network) using Wi-Fi 6.

Furthermore, for example, the communication device 60 communicates with the information processing device 200 and remote operation support device 300 outside the work site via a wide area communication line that includes the work site, i.e., a wide area network (WAN). Wide area networks include, for example, wide area mobile communication networks, satellite communication networks, and the Internet.

<Control System>
4, the control system of the shovel 100 includes a controller 30. The control system of the shovel 100 according to this embodiment also includes an operating pressure sensor 29, an imaging device 40, and sensors S1 to S5.

The controller 30 performs various controls related to the excavator 100.

The functions of the controller 30 may be realized by any hardware or any combination of hardware and software. For example, as shown in FIG. 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, all connected via a bus B1.

The auxiliary storage device 30A is a non-volatile storage device that stores the programs to be installed as well as necessary files and data. The auxiliary storage device 30A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or flash memory.

For example, when a program startup command is received, memory device 30B loads the program from auxiliary storage device 30A so that it can be read by CPU 30C. Memory device 30B is, for example, SRAM (Static Random Access Memory).

The CPU 30C, for example, executes programs loaded into the memory device 30B and implements various functions of the controller 30 according to the program's instructions.

The interface device 30D functions, for example, as a communication interface for connecting to a communication line inside the excavator 100. The interface device 30D may include multiple different types of communication interfaces to match the type of communication line to be connected.

The interface device 30D also functions as an external interface for reading data from and writing data to a recording medium. The recording medium is, for example, a dedicated tool connected via a detachable cable to a connector installed inside the cabin 10. The recording medium may also be a general-purpose recording medium, such as an SD memory card or USB (Universal Serial Bus) memory. As a result, programs that realize the various functions of the controller 30 can be provided, for example, by a portable recording medium and installed in the auxiliary storage device 30A of the controller 30. The programs may also be downloaded from another computer external to the excavator 100 via the communication device 60 and installed in the auxiliary storage device 30A.

Note that some of the functions of controller 30 may be realized by other controllers (control devices). In other words, the functions of controller 30 may be realized in a distributed manner by multiple controllers.

The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, i.e., the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The pilot pressure detection signal from the operating pressure sensor 29 corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 is input to the controller 30.

If the operating device 26 is electrical, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal received from the operating device 26.

The imaging device 40 acquires images of the area around the shovel 100. Furthermore, the imaging device 40 may acquire (generate) three-dimensional data (hereinafter simply referred to as "three-dimensional data of objects") representing the position and outline of objects around the shovel 100 within the imaging range (angle of view) based on the acquired images and distance-related data described below. The three-dimensional data of objects around the shovel 100 is, for example, coordinate information data of a point cloud representing the surface of the object, distance image data, etc.

For example, as shown in FIG. 2 , the imaging device 40 includes a camera 40F that captures images in front of the upper rotating body 3, a camera 40B that captures images behind the upper rotating body 3, a camera 40L that captures images to the left of the upper rotating body 3, and a camera 40R that captures images to the right of the upper rotating body 3. This allows the imaging device 40 to capture images of the entire circumference of the excavator 100, i.e., a 360-degree angular range, from the top view of the excavator 100. Furthermore, the operator can view the images captured by cameras 40B, 40L, and 40R and peripheral images such as processing images generated based on the captured images via the output device 50 (display device 50A) and the remote control display device, thereby checking the conditions to the left, right, and rear of the upper rotating body 3. Furthermore, by viewing the images captured by camera 40F and peripheral images such as processing images generated based on the captured images via the remote control display device, the operator can remotely operate the excavator 100 while checking the operation of the attachment AT, including the bucket 6. Hereinafter, cameras 40F, 40B, 40L, and 40R may be referred to collectively or individually as "camera 40X."

Camera 40X is, for example, a monocular camera. Camera 40X may also be capable of acquiring distance (depth) data in addition to two-dimensional images, such as a stereo camera or a TOF (Time Of Flight) camera (hereinafter collectively referred to as a "3D camera").

The output data of the imaging device 40 (camera 40X) (e.g., image data, three-dimensional data of objects around the shovel 100, etc.) is input into the controller 30 via a one-to-one communication line or an on-board network. This allows the controller 30 to, for example, monitor objects around the shovel 100 based on the output data of the camera 40X. Furthermore, for example, the controller 30 can determine the environment around the shovel 100 based on the output data of the camera 40X. Furthermore, for example, the controller 30 can determine the posture of the attachment AT shown in the captured image based on the output data of the camera 40X (camera 40F). Furthermore, for example, the controller 30 can determine the posture of the body (upper rotating body 3) of the shovel 100 based on the objects around the shovel 100 based on the output data of the camera 40X.

Note that some of the cameras 40F, 40B, 40L, and 40R may be omitted. For example, if the shovel 100 is not remotely operated, the cameras 40F and 40L may be omitted. This is because the situation in front of and to the left of the shovel 100 is relatively easy to see from the operator in the cabin 10. Furthermore, instead of or in addition to the imaging device 40 (camera 40X), a distance sensor may be provided on the upper rotating body 3. The distance sensor is attached, for example, to the top of the upper rotating body 3, and acquires data regarding the distance and direction of surrounding objects relative to the shovel 100. Furthermore, the distance sensor may acquire (generate) three-dimensional data (e.g., point cloud coordinate information data) of objects around the shovel 100 within its sensing range based on the acquired data. The distance sensor is, for example, a LIDAR (Light Detection and Ranging) sensor. Furthermore, the distance sensor may be, for example, a millimeter-wave radar, an ultrasonic sensor, an infrared sensor, or the like.

Sensor S1 is attached to the boom 4 and detects the attitude angle (hereinafter referred to as the "boom angle") around the rotation axis of the base end of the boom 4, which corresponds to the connection point with the upper rotating body 3. Sensor S1 includes, for example, a rotary potentiometer, rotary encoder, acceleration sensor, angular acceleration sensor, 6-axis sensor, IMU (Inertial Measurement Unit), etc. The same may apply to sensors S2 and S4 below. Sensor S1 may also include a cylinder sensor that detects the extension/retraction position of the boom cylinder 7. The same may apply to sensor S2 below. The boom angle detection signal from sensor S1 is input to controller 30. This allows controller 30 to grasp the attitude state of the boom 4.

Sensor S2 is attached to arm 5 and detects the posture angle (hereinafter referred to as "arm angle") around the rotation axis of the base end of arm 5, which corresponds to the connection point between arm 5 and boom 4. The arm angle detection signal from sensor S2 is input to controller 30. This allows controller 30 to grasp the posture state of arm 5.

Sensor S3 is attached to the bucket 6 and detects the attitude angle (hereinafter referred to as the "arm angle") around the rotation axis of the base end of the bucket 6, which corresponds to the connection point with the arm 5. The arm angle detection signal from sensor S3 is input to controller 30. This allows controller 30 to grasp the attitude state of the bucket 6.

Sensor S4 detects the inclination state of the machine body (e.g., the upper rotating body 3) relative to a predetermined reference plane (e.g., a horizontal plane). Sensor S4 is attached, for example, to the upper rotating body 3 and detects the inclination angles (hereinafter referred to as the "fore-aft inclination angle" and the "left-right inclination angle") of the excavator 100 (i.e., the upper rotating body 3) about two axes in the fore-aft and left-right directions. The detection signals corresponding to the inclination angles (fore-aft inclination angle and left-right inclination angle) detected by sensor S4 are input to controller 30. This allows controller 30 to grasp the inclination state of the machine body (upper rotating body 3).

Sensor S5 is attached to the upper rotating body 3 and outputs detection information related to the rotation state of the upper rotating body 3. Sensor S5 detects, for example, the rotation angular velocity and rotation angle of the upper rotating body 3. Sensor S5 includes, for example, a gyro sensor, resolver, rotary encoder, etc. The detection information related to the rotation state detected by sensor S5 is input into controller 30. This allows controller 30 to grasp the rotation state, such as the rotation angle, of the upper rotating body 3.

If sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like that can detect angular velocity around three axes, the rotation state of the upper rotating body 3 (e.g., rotation angular velocity) may be detected based on the detection signal of sensor S4. In this case, sensor S5 may be omitted. Also, if it is possible to determine the posture state of the upper rotating body 3, attachment AT, etc. based on the output of the imaging device 40 or distance sensor, at least some of sensors S1 to S5 may be omitted.

<Hardware configuration of information processing device>
FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 200.

The functions of the information processing device 200 are realized by any hardware or any combination of hardware and software. For example, as shown in FIG. 5, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed calculation device 205, a communication interface 206, an input device 207, and a display device 208, which are connected via a bus B2.

The external interface 201 functions as an interface for reading data from and writing data to the recording medium 201A. Examples of the recording medium 201A include flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), BDs (Blu-ray (registered trademark) Discs), SD memory cards, USB memory, etc. This allows the information processing device 200 to read various data used in processing through the recording medium 201A, store the data in the auxiliary storage device 202, and install programs that realize various functions.

In addition, the information processing device 200 may obtain various data and programs used in processing from external devices via the communication interface 206.

The auxiliary storage device 202 stores various installed programs as well as files and data required for various processes. Examples of the auxiliary storage device 202 include a hard disk drive (HDD), a solid state disk (SSD), and flash memory.

When a program startup instruction is received, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, DRAM (Dynamic Random Access Memory) and SRAM.

The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and realizes various functions related to the information processing device 200 in accordance with the programs.

The high-speed calculation device 205 works in conjunction with the CPU 204 to perform calculations at a relatively high speed. The high-speed calculation device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc.

Note that the high-speed calculation device 205 may be omitted depending on the required calculation processing speed.

The communication interface 206 is used as an interface for connecting to an external device so that communication is possible. This allows the information processing device 200 to communicate with external devices such as the excavator 100 through the communication interface 206. The communication interface 206 may also have multiple types of communication interfaces depending on the communication method between the connected devices, etc.

The input device 207 accepts various inputs from the user.

The input device 207 includes, for example, an operation input device that accepts mechanical operation input from the user. The operation input device includes, for example, a button, a toggle, a lever, etc. The operation input device also includes, for example, a touch panel implemented in the display device 208, a touch pad provided separately from the display device 208, etc.

The input device 207 also includes, for example, an audio input device capable of accepting audio input from the user. The audio input device includes, for example, a microphone capable of collecting the user's voice.

The input device 207 also includes, for example, a gesture input device capable of accepting gesture input from a user. The gesture input device includes, for example, a camera capable of capturing images of the user's gestures.

The input device 207 also includes, for example, a biometric input device capable of accepting biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information about the user's fingerprint or iris.

The display device 208 displays information screens and operation screens to the user. For example, the display device 208 includes the remote control display device described above. The display device 208 is, for example, a liquid crystal display or an organic EL (electroluminescence) display.

Like the information processing device 200, the remote operation support device 300 may be realized by any hardware or any combination of hardware and software, and may employ a similar hardware configuration. For example, like the information processing device 200 ( FIG. 6 ), the remote operation support device 300 is primarily composed of a computer including a CPU, memory device, auxiliary storage device, interface device, input device, and display device. The memory device is, for example, SRAM or DRAM. The auxiliary storage device is, for example, an HDD, SSD, EEPROM, or flash memory. The interface device includes an external interface for connecting to an external recording medium and a communication interface for communicating with external devices such as the shovel 100. The input device includes, for example, a lever-type operation input device. This allows the operator to use the operation input device to input operations related to the actuators of the shovel 100, and the remote operation support device 300 can transmit signals corresponding to the operation input to the shovel 100 using the communication interface. This allows the operator to remotely control the shovel 100 using the remote operation support device.

[Functional configuration for generating the trajectory of the working part]
Next, with reference to FIGS. 6 to 9 in addition to FIGS. 1 to 5, a functional configuration relating to generation of a trajectory of a working portion of the shovel 100 (hereinafter referred to as a "target trajectory") will be described.

Figure 6 is a block diagram showing a first example of a functional configuration related to the generation of a target trajectory for the working part of the shovel 100. Figure 7 is a diagram showing an example of a screen (screen 700) related to the generation of a target trajectory for the working part of the shovel 100, displayed on the display device 50A. Figure 8 is a diagram showing another example of a screen (screen 800) related to the generation of a target trajectory for the working part of the shovel 100, displayed on the display device 50A. Figure 9 is a diagram showing yet another example of a screen (screen 900) related to the generation of a target trajectory for the working part of the shovel 100, displayed on the display device 50A.

When the excavator 100 is remotely operated, screens similar to those shown in Figures 7 and 8 are displayed on the remote operation support device 300 (display device).

The working part of the excavator 100 is, for example, the tip or back of the bucket 6.

The shovel 100 includes a support device 150. The support device 150 provides support for the work of the shovel 100.

As shown in FIG. 6, the support device 150 includes an operation device 26, a controller 30, an imaging device 40, and an output device 50. In addition, if the shovel 100 is remotely operated, the support device 150 may also include a communication device 60.

The controller 30 includes, as functional units, an operation log providing unit 301 and a work support unit 302.

Note that if the operation support system SYS includes multiple shovels 100, there may be shovels 100 whose controller 30 includes only the operation log providing unit 301 and the work support unit 302, and shovels 100 whose controller 30 includes only the latter. In this case, the former shovel 100 only has the function of acquiring the operation log of the shovel 100 and providing it to the information processing device 200, which is used for the operator's operation support function (operation suggestion function) in the latter shovel 100.

The information processing device 200 includes, as functional units, an operation log acquisition unit 2001, an operation log storage unit 2002, a training data generation unit 2003, a machine learning unit 2004, a trained model storage unit 2005, and a distribution unit 2006.

The operation log providing unit 301 is a functional unit that acquires the operation log of the shovel 100, which is the original data used to realize the function of generating the target trajectory of the working part of the shovel 100, and provides it to the information processing device 200. Specifically, it acquires the operation log when the shovel 100 is operated by an operator with a long history of operating the shovel 100 and a relatively high level of experience (hereinafter referred to as an "expert" for convenience), and provides it to the information processing device 200.

The operation log of the shovel 100 includes data regarding the shape of the work object around the shovel 100 and data regarding the operation of the shovel 100 performed on the shape of the work object. The data regarding the shape of the work object around the shovel 100 is, for example, data regarding the topographical shape of the ground at the work site as the work object of the shovel 100. The data regarding the shape of the work object of the shovel 100 is, for example, image data from the imaging device 40 or three-dimensional data of the work object obtained from that image data. The data regarding the operation of the shovel 100 is, for example, data representing the operation content of the operator. The data representing the operation content of the operator is, for example, output data from the operating pressure sensor 29 in the case of a hydraulic pilot-type operating device 26, or output data (operation signal data) of the operating device 26 in the case of an electric operating device 26. The data regarding the operation of the shovel 100 may also be data representing the operating state of the shovel 100 actually performed in response to the operation of the operator. Data representing the operating state of the shovel 100 is, for example, output data from sensors S1 to S5, or data relating to the posture state of the shovel 100 obtained from the output data from sensors S1 to S5.

The operation log providing unit 301 includes an operation log recording unit 301A, an operation log storage unit 301B, and an operation log transmission unit 301C.

The operation log recording unit 301A acquires the operation log of the shovel 100 and records it in the operation log storage unit 301B. For example, each time the shovel 100 performs an operation, the operation log recording unit 301A records data related to the shape of the work object around the shovel 100 at the start of or immediately before the operation, and data related to the operation of the shovel 100, in the operation log storage unit 301B.

The operation log storage unit 301B stores an accumulated operation log of the shovel 100. For example, the operation log storage unit 301B stores data relating to the shape of the work object around the shovel 100 for each operation of the shovel 100 and data relating to the operation of the shovel 100 in a linked manner. Specifically, the operation log storage unit 301B may store record data indicating the correspondence between the data relating to the shape of the work object around the shovel 100 for each operation of the shovel 100 and the data relating to the operation of the shovel 100, and a database of operation logs may be constructed.

Note that the operation log stored in the operation log storage unit 301B that has already been transmitted to the information processing device 200 by the operation log transmission unit 301C (described below) may be deleted afterward.

The operation log transmission unit 301C transmits the operation log of the shovel 100 stored in the operation log memory unit 301B to the information processing device 200 via the communication device 60. The operation log transmission unit 301C may also transmit to the information processing device 200 data relating to the shape of the work object around the shovel 100 for each operation of the shovel 100, and record data indicating the correspondence between data relating to the operation of the shovel 100.

For example, the operation log transmission unit 301C transmits an untransmitted operation log of the shovel 100 stored in the operation log storage unit 301B to the information processing device 200 in response to a signal (hereinafter referred to as a "transmission request signal") received from the information processing device 200 requesting transmission of the operation log of the shovel 100. Furthermore, the operation log transmission unit 301C may automatically transmit an untransmitted operation log of the shovel 100 stored in the operation log storage unit 301B to the information processing device 200 at a predetermined timing. The predetermined timing is, for example, when the shovel 100 stops operating (key switch is turned off) or starts operating (key switch is turned on).

The operation log acquisition unit 2001 acquires the operation log of the excavator 100 received from the excavator 100.

The operation log acquisition unit 2001 acquires the operation log of the shovel 100 by transmitting a transmission request signal to the shovel 100 in response to an operation by the user of the information processing device 200, or automatically at a predetermined timing. The operation log acquisition unit 2001 may also acquire the operation log of the shovel 100 transmitted from the shovel 100 at a predetermined timing.

The operation log storage unit 2002 stores the operation log of the shovel 100 acquired by the operation log acquisition unit 2001 in an accumulated form. For example, similar to the case of the operation log storage unit 301B, the operation log storage unit 2002 stores data relating to the shape of the work target around the shovel 100 for each operation of the shovel 100 in a linked form with data relating to the operation of the shovel 100.

The training data generation unit 2003 generates training data for machine learning based on the operation log of the shovel 100 stored in the operation log storage unit 2002. The training data generation unit 2003 may generate training data automatically by batch processing, or may generate training data in response to input from a user of the information processing device 200. The training data is a combination of data relating to the shape of the work object around the shovel 100 as input data and data representing the trajectory (track) of the working part of the shovel 100 corresponding to the input data as correct output data (hereinafter referred to as "correct data").

Data representing the trajectory of the working part of the shovel 100 is generated, for example, based on the output data of sensors S1 to S5 included in data related to the operation of the shovel 100.

The machine learning unit 2004 performs machine learning on the base learning model based on the set of training data generated by the training data generation unit 2003, and generates a trained model LM. The trained model LM (base learning model) includes, for example, a neural network such as a DNN (Deep Neural Network).

The learned model LM uses, for example, data regarding the type of operation of the shovel 100 and the shape of a work target around the shovel 100 as input conditions, and outputs data representing a target trajectory of the working part of the shovel 100 and a prediction probability. The learned model LM also outputs data representing the target trajectory of the working part of the shovel 100 and types of operation of the shovel 100, including, for example, an excavation operation, a sweeping operation, a horizontal dragging operation, a compaction operation, a broom operation, etc. The sweeping operation is, for example, an operation in which the attachment AT is operated to push the bucket 6 forward along the ground, thereby sweeping soil and sand forward with the back surface of the bucket 6. In the sweeping operation, for example, the attachment AT lowers the boom 4 and opens the arm 5. The horizontal dragging operation is, for example, an operation in which the attachment AT is operated to move the tip of the bucket 6 approximately horizontally along the ground, pulling it toward the user, thereby leveling unevenness on the ground surface. In the horizontal pulling operation, for example, the attachment AT raises the boom 4 and closes the arm 5. The compaction operation is, for example, an operation in which the attachment AT is operated to press the ground with the back surface of the bucket 6. Alternatively, the compaction operation may be an operation in which the bucket 6 is pushed forward along the ground, the back surface of the bucket 6 sweeps soil and sand to a predetermined position forward, and then the ground at the predetermined position is pressed against the ground with the back surface of the bucket 6. In the compaction operation, for example, the attachment AT lowers the boom 4 while pressing the ground. The broom operation is, for example, an operation in which the upper rotating body 3 is operated to rotate the bucket 6 left and right while keeping it along the ground. Alternatively, the broom operation may be an operation in which the attachment AT and the upper rotating body 3 are operated to push the bucket 6 forward while alternately rotating it left and right while keeping it along the ground. In the broom operation, for example, the upper rotating body 3 alternately rotates left and right. Furthermore, in the broom operation, for example, in addition to the alternating left and right rotation of the upper rotating body 3, the boom 4 may be lowered and the arm 5 may be opened, as in the sweeping operation. The prediction probability represents the reliability of the target trajectory of the work part. As described above, the learned model LM reflects the operation log of the skilled worker operating the shovel 100, and the higher the prediction probability, the higher the reliability of the target trajectory of the work part. The prediction probability also represents the degree of conformance of the target trajectory of the work part with the shape of the work object around the shovel 100, which is an input condition. The higher the prediction probability, the more likely the skilled worker is to determine that the candidate operation is appropriate for the shape of the work object. For example, the learned model LM is generated for each operation, such as ground leveling, slope construction, and embankment work.

The learned model memory unit 2005 stores the learned model LM output by the machine learning unit 2004.

The distribution unit 2006 distributes the trained model LM to the excavator 100.

For example, when a trained model LM is generated by the machine learning unit 2004, the distribution unit 2006 distributes the most recently generated trained model LM to the shovel 100. Furthermore, the distribution unit 2006 may distribute the latest trained model LM in the trained model memory unit 2005 to the shovel 100 in response to a signal received from the shovel 100 requesting distribution of the trained model LM.

The work support unit 302 is a functional unit that supports the work of the excavator 100 as operated by the operator.

The work support unit 302 includes a trained model storage unit 302A, a work object shape acquisition unit 302B, an action selection unit 302C, a condition setting unit 302D, a trajectory generation unit 302E, a display processing unit 302F, and an action control unit 302G.

The learned model storage unit 302A stores the learned model LM distributed from the information processing device 200 and received via the communication device 60.

The work object shape acquisition unit 302B acquires data regarding the shape of the work object (terrain shape) around the excavator 100 based on the output of the imaging device 40 and distance sensor.

The operation selection unit 302C selects an operation (type) of the shovel 100 from among a plurality of operation candidates in response to input from the user (operator) received via the input device 52. Furthermore, when the shovel 100 is remotely operated, the operation selection unit 302C may select an operation of the shovel 100 from among a plurality of operation candidates in response to input from the user (operator) using the remote operation assistance device 300 received via the communication device 60.

The condition setting unit 302D sets preconditions for generating a target trajectory for the working part of the shovel 100 in response to input from the user (operator) received through the input device 52. When the shovel 100 is remotely operated, the condition setting unit 302D may set preconditions for the target trajectory of the shovel 100 in response to input from the user (operator) using the remote operation support device 300 received through the communication device 60. The condition setting unit 302D may also set preconditions automatically, without relying on input from the user. For example, the condition setting unit 302D may automatically set preconditions based on a trained model generated using, as a training dataset, a data history of combinations of data related to the shape of the work object and preconditions set for the shape of the work object. In this case, the condition setting unit 302D may modify the automatically set preconditions in response to input from the user.

The precondition is, for example, a point (hereinafter referred to as "target point") on the topographical shape around the shovel 100 that serves as a target when the shovel 100 is operating. Target points include, for example, a target point through which the working part passes when the shovel 100 is operating, or a point corresponding to a location where the bucket 6 unloads soil when the shovel 100 is operating. The precondition may also include the attitude state of the bucket 6 at the target point (the attitude angle of the bucket 6).

The trajectory generation unit 302E generates a target trajectory for the work portion of the shovel 100 based on the data acquired by the work object shape acquisition unit 302B, the target shape of the work object, the operations selected by the operation selection unit 302C, and the preconditions set by the condition setting unit 302D. The target shape of the work object is, for example, a target construction surface representing a flat or curved surface as a construction object formed by work on the work object (the ground at the work site). The target shape of the work object is set by inputting parameters representing a flat or curved surface from the user, for example, via the input device 52 or the remote operation support device 300 (input device). The target shape of the work object may also be distributed to the shovel 100 from an external device such as the information processing device 200. The trajectory generation unit 302E applies the learned model LM to the data acquired by the work object shape acquisition unit 302B, the target shape of the work object, the operations selected by the operation selection unit 302C, and the preconditions set by the condition setting unit 302D as input data. The trajectory generation unit 302E may also output a target trajectory for the work part from the learned model LM using the target shape of the work object, the action selected by the action selection unit 302C, and the data acquired by the work object shape acquisition unit 302B as input data. The trajectory generation unit 302E may then generate a target trajectory for the work part by optimizing the output target trajectory for the work part using the prerequisites set by the condition setting unit 302D.

The display processing unit 302F causes the display device 50A to display a screen related to the generation of a target trajectory for the working portion of the shovel 100 (see FIGS. 7 and 8 ). The screen related to the generation of a target trajectory for the working portion of the shovel 100 includes, for example, an operation screen that allows the user (operator) to perform operation input related to the operation of the shovel 100 selected by the operation selection unit 302C and the prerequisites set by the condition setting unit 302D. The screen related to the generation of a target trajectory for the working portion of the shovel 100 also includes a screen that displays the target trajectory for the working portion of the shovel 100, generated by the trajectory generation unit 302E. When the shovel 100 is remotely operated, the display processing unit 302F may transmit data related to the screen related to the generation of the target trajectory for the working portion of the shovel 100 to the remote operation assistance device 300 via the communication device 60. This allows the display processing unit 302F to display a screen related to the generation of a target trajectory for the working portion of the shovel 100 on the remote operation assistance device 300 (display device).

For example, as shown in Figures 7 and 8, the display processing unit 302F displays screens 700 and 800 on the display device 50A.

As shown in Figure 7, screen 700 includes images TG, CG, SB, and PB1.

Image TG is an image representing the shape of the terrain around the shovel 100. Image TG is generated based on data acquired by the work object shape acquisition unit 302B. In this example, image TG is an image representing the shape of the terrain around the shovel 100 as seen from a specified viewpoint outside the shovel 100. The specified viewpoint can be changed, for example, in response to input from the user (operator) via the input device 52 or the remote operation support device 300 (input device).

Image CG is an image representing the shovel 100.

The positional relationship between the images TG and CG is set to be the same as the actual positional relationship between the terrain shape around the shovel 100 and the shovel 100.

Image SB represents candidate actions that can be selected by the action selection unit 302C. In this example, image SB includes images SB1 to SB5 that represent candidate actions of the shovel 100 that can be performed during ground leveling work.

Image SB1 is an operation icon that allows the user to select a combination of excavation and soil removal operations for the excavator 100.

Image SB2 is an operation icon that allows the user to select the sweeping operation of the shovel 100.

Image SB3 is an operation icon that allows the user to select the horizontal pulling operation of the shovel 100.

Image SB4 is an operation icon that allows the user to select the broom operation of the shovel 100.

Image SB5 is an operation icon that allows the user to select the compaction operation of the shovel 100.

The user can specify one of images SB1 to SB5 via the input device 52 or remote operation support device 300 (input device), and select an operation for the shovel 100 via the operation selection unit 302C. In this example, the cursor (the textured area in the figure) is located on image SB1, indicating that a combination of excavation and earth removal operations for the shovel 100 has been selected.

In addition to images SB1 to SB5, image SB may display an operation icon that allows the user to select an action other than the actions corresponding to images SB1 to SB5. Furthermore, image SB may display an operation icon that allows the user to select an action other than the actions corresponding to images SB1 to SB5, instead of at least one of images SB1 to SB5.

In this example, image TG includes image areas TG1 and TG2.

Image area TG1 represents the protrusions on the ground around (in front of) the shovel 100.

Image area TG2 represents a depression in the ground around (in front of) the shovel 100.

In this example, screen 700 also includes images P1 and P2 corresponding to target points.

Image P1 is displayed superimposed on image area TG1.

Image P2 is displayed superimposed on image area TG2.

For example, the user can specify image areas TG1 and TG2 via the input device 52 or the remote operation support device 300 (input device) to set target points corresponding to images P1 and P2 via the condition setting unit 302D. The user may be able to set a target point within the entire range of the image TG via the input device 52 or the remote operation support device 300 (input device), or may be able to set a target point limited to the range within the entire range of the image TG that the working portion of the bucket 6 can reach. In the former case, if a target point is set within the range within the entire range of the image TG that the working portion of the bucket 6 can reach, a display indicating an error (warning) may be displayed on the screen 700. In the latter case, an image representing the range within the entire range of the image TG that the working portion of the bucket 6 can reach may be displayed superimposed on the image TG. The user may also be able to delete a set target point via the input device 52 or the remote operation support device 300 (input device).

In addition, in this example, images RC1 and RC2 are displayed on screen 700 in association with images P1 and P2, respectively.

Image RC1 is an image that shows the preconditions for the bucket 6 attitude angle corresponding to the target point corresponding to image P1.

Image RC2 shows the preconditions for the bucket 6 attitude angle corresponding to the target point corresponding to image P2.

For example, by specifying images P1 and P2 via the input device 52 or the remote operation support device 300 (input device), the user can set preconditions for the attitude angle of the bucket 6 corresponding to images RC1 and RC2 via the condition setting unit 302D.

Image PB1 is an operation icon that causes the trajectory generation unit 302E to generate a trajectory for the working portion of the bucket 6 in accordance with the operation selected on screen 800 and the prerequisites set on screen 800.

For example, the user can generate a target trajectory for the bucket 6 via the trajectory generation unit 302E by manipulating the image PB1 via the input device 52 or the remote operation support device 300 (input device).

When image PB1 is operated, the display content of display device 50A transitions from screen 700 to screen 800.

Like screen 700, screen 800 includes images TG, CG, and SB. Also, like screen 700, screen 800 includes images P1 and P2. Also, screen 800 includes images OG, CG1, and PB2.

Image OG is an image showing the target trajectory.

Image CG1 is an image showing bucket 6, displayed in association with image OG, which corresponds to the target trajectory.

In this example, image OG represents a target trajectory that realizes an operation in which soil and sand are scooped up at a target point corresponding to image P1 using an excavation operation and dumped at a target point corresponding to image P2. Image OG may be displayed so that the portion of the trajectory where the working part of the bucket 6 comes into contact with the soil and the other portion of the trajectory can be distinguished. For example, image OG displays the portion of the trajectory where the working part of the bucket 6 comes into contact with the soil and the other portion of the trajectory in different colors.

Image PB2 is an operation icon for reproducing on screen 800, as a moving image (animation), the operation of moving the working part of the bucket 6 along the target trajectory corresponding to image OG.

For example, by operating image PB2 via input device 52 or remote operation support device 300 (input device), the user can display on screen 800 a moving image in which image CG1, which corresponds to bucket 6, moves along image OG, which corresponds to the target trajectory. Therefore, by checking the moving image, the user can determine whether the target trajectory is appropriate.

The moving image may also display the shape of the work object (terrain shape) after the operation of the shovel 100 to move the bucket 6 along the target trajectory. In other words, the screen 800 may display the predicted shape of the work object (terrain shape) around the shovel 100 after the shovel 100 is operated to move the bucket 6 along the target trajectory corresponding to image OG. This allows the user to more appropriately determine whether the target trajectory is appropriate by checking the moving image and the predicted changes in the terrain shape.

When image PB1 is operated, the display content of display device 50A transitions from screen 800 to screen 900.

Like screen 800, screen 900 includes images TG, CG, and SB. Also, like screen 800, screen 900 includes images P1 and P2. Also, screen 900 includes images OG and CG1. Also, screen 900 includes image PB3.

Image PB3 is an operation icon for automatically operating the excavator 100 so as to move the working portion of the bucket 6 along the target trajectory corresponding to image OG.

For example, by operating image PB3 via the input device 52 or remote operation support device 300 (input device), the user can automatically operate the excavator 100 via the operation control unit 302G so that the bucket 6 moves along the target trajectory corresponding to image OG.

It may also be possible for the user to automatically operate the shovel 100 so that the bucket 6 moves along the target trajectory corresponding to image OG, without checking the above-mentioned moving image. In this case, in addition to image PB2, an operation icon corresponding to image PB3 is displayed on the screen 800.

Returning to Figure 7, the operation control unit 302G operates the shovel 100 in response to input from the user (operator) received through the input device 52, so that the working portion of the bucket 6 moves along the target trajectory generated by the trajectory generation unit 302E. Specifically, the operation control unit 302G controls the hydraulic control valve 31 while determining the position of the working portion of the bucket 6 from the outputs of sensors S1 to S5, etc., thereby operating the shovel 100 so that the working portion of the bucket 6 moves along the target trajectory.

For example, in response to input from the user an instruction to execute an operation, the operation control unit 302G operates the shovel 100 so that the working portion of the bucket 6 moves along the target trajectory generated by the trajectory generation unit 302E.

In addition, the operation control unit 302G may operate the shovel 100 in response to the operation of the operating device 26 or a remote control signal, thereby assisting the operator in operating the shovel 100 so that the working portion of the bucket 6 moves along the target trajectory generated by the trajectory generation unit 302E.

[Processing related to generation of trajectory of working part]
Next, with reference to FIG. 10, a process for generating a target trajectory for the working portion of the shovel 100 will be described.

Figure 10 is a flowchart that outlines an example of a process for generating a target trajectory for the working part of the shovel 100.

The flowchart in Figure 10 is executed repeatedly, for example, during operation of a function related to generating a target trajectory for the working part of the shovel 100. The function related to generating a target trajectory for the working part of the shovel 100 is activated (started) by inputting instructions from the user received through the input device 52 or the remote operation support device 300 (input device).

As shown in FIG. 10, in step S102 (an example of an acquisition step), the work object shape acquisition unit 302B acquires data regarding the shape of the work object around the shovel 100 from the imaging device 40.

When the processing of step S102 is completed, the controller 30 proceeds to step S104.

In step S104 (an example of a display step), the display processing unit 302F displays a setting screen (e.g., screen 700) including an image representing the terrain shape on the display device 50A or the remote operation support device 300 (display device) based on the data acquired in step S102.

When the processing of step S104 is completed, the controller 30 proceeds to step S106.

In step S106, the operation selection unit 302C selects one operation from among multiple candidate operations for the shovel 100 in response to input from the user.

When the processing of step S106 is completed, the controller 30 proceeds to step S108.

In step S108, the condition setting unit 302D (an example of a setting step) sets preconditions for generating the target trajectory of the working part of the excavator 100 in accordance with input from the user.

When the processing of step S108 is completed, the controller 30 proceeds to step S110.

Note that the order of steps S106 and S108 may change depending on input from the user.

In step S110 (an example of a generation step), the trajectory generation unit 302E generates a target trajectory for the working part of the excavator 100 for the operation selected in step S106 under the preconditions set in step S108.

When the processing of step S110 is completed, the controller 30 proceeds to step S112.

In step S112, the display processing unit 302F displays an image representing the target trajectory generated in step S110 on the display device 50A or the remote operation assistance device 300 (display device).

When the processing of step S112 is completed, the controller 30 proceeds to step S114.

In step S114, the controller 30 determines whether or not an operation input has been received that instructs the shovel 100 to perform an operation to move the working portion of the bucket 6 along the target trajectory generated in step S112. If an operation input instructing the shovel 100 to perform an operation has been received, the controller 30 proceeds to step S116; if any other operation, specifically an operation to generate the target trajectory again, has been received, the controller 30 returns to step S106.

In step S116, the operation control unit 302G controls the hydraulic control valve 31 to automatically operate the excavator 100 so that the working portion of the bucket 6 moves along the target trajectory generated in the most recent processing of step S110.

When the processing of step S116 is completed, the controller 30 ends the processing of this flowchart.

Note that upon completion of the processing of step S116, the excavator 100 (attachment AT) may be in a state where the working portion of the bucket 6 is at the end point of the target trajectory, or may have returned to the posture state it had before the processing of step S114 began.

In this way, in this example, the support device 150 (controller 30) can generate a target trajectory that matches the shape of the work object. This can improve the work efficiency of the excavator 100.

In addition, in this example, the support device 150 can generate a target trajectory that conforms to prerequisites such as the target point and the attitude angle of the bucket 6. Therefore, a more appropriate target trajectory can be generated that reflects the user's judgment and intentions, which are based on the shape of the work object, etc.

Furthermore, in this example, the support device 150 can automatically operate the shovel 100 so that the working portion of the bucket 6 moves along the generated target trajectory. Therefore, even an inexperienced operator can cause the shovel 100 to perform appropriate operations, thereby further improving the work efficiency of the shovel 100.

Other Embodiments
Next, another embodiment will be described.

The above-described embodiments may be combined, modified, or altered as appropriate.

For example, in the above-described embodiment, the trajectory generation unit 302E may generate a target trajectory without using the learned model LM. For example, a reference trajectory for the working part may be defined in advance for each of a plurality of candidate actions, and the trajectory generation unit 302E may generate a target trajectory for the working part by optimizing the reference trajectory for the action selected by the action selection unit 302C in accordance with data related to the shape of the work target (terrain shape) around the shovel 100 and preconditions.

Furthermore, in the above-described embodiment and its variations, data regarding the shape of the work object around the shovel 100 may be acquired based on data from an imaging device, distance sensor, or the like installed outside the shovel 100. For example, data from an imaging device or distance sensor installed at the work site is received by the shovel 100 via the communication device 60, allowing the work object shape acquisition unit 302B to acquire data regarding the shape of the work object around the shovel 100. Furthermore, for example, data from an imaging device or distance sensor mounted on a drone flying above the work site is received by the shovel 100 via the communication device 60, allowing the work object shape acquisition unit 302B to acquire data regarding the shape of the work object around the shovel 100.

Furthermore, in the above-described embodiment and its variations, some or all of the functions of the assistance device 150 may be transferred to the remote operation assistance device 300.

Furthermore, in the above-described embodiment and its variations, some or all of the functions of the assistance device 150 may be transferred to the information processing device 200.

[Effect]
Next, the operation of the operation support system SYS (operation support device) according to this embodiment will be described.

In this embodiment, the support device includes an acquisition unit, a display unit, a setting unit, and a generation unit. The support device is, for example, the support device 150. The acquisition unit is, for example, the work object shape acquisition unit 302B. The display unit is, for example, the display device 50A. The setting unit is, for example, the condition setting unit 302D. The generation unit is, for example, the trajectory generation unit 302E. Specifically, the acquisition unit acquires data related to the terrain shape of the construction object around the work machine. The work machine is, for example, an excavator 100. The display unit displays an image representing the terrain shape of the construction object based on the data acquired by the acquisition unit. The setting unit sets a point (target point) on the terrain shape of the construction object that will serve as a target for the work machine when it operates. The generation unit generates a trajectory for the work portion of the work machine based on the data acquired by the acquisition unit, the target shape of the construction object, and the point set by the setting unit.

This makes it easier to generate the trajectory of the work part of the work machine. In addition, because a target point is set, it is possible to generate a more appropriate trajectory of the work part that reflects, for example, the judgment and intention of the user who visually recognizes the shape of the work object around the work machine. This therefore improves the work efficiency of the work machine.

In this embodiment, the assistance device may also include a selection unit. Specifically, the selection unit may select one operation from among multiple candidate operations of the work machine in response to input from the user. The generation unit may then generate a trajectory of the working part resulting from one operation of the work machine based on the data acquired by the acquisition unit and the points set by the setting unit.

This allows the support device to define the operation of the work machine and generate the trajectory of the working part, thereby further improving the work efficiency of the work machine.

In addition, in this embodiment, the setting unit may set a target point (target point) on the terrain shape around the work machine that will serve as a target when the work machine is operating, and the posture of the work part that corresponds to that point, in response to input from the user.

This allows the assistance device to generate a more appropriate trajectory for the working part that reflects the user's judgment and intention regarding the posture of the working part. This can further improve the work efficiency of the work machine.

In addition, in this embodiment, the display unit may display an image representing the trajectory generated by the generation unit, superimposed on an image representing the terrain shape around the work machine.

This allows the user to visually check the generated image. Furthermore, by simultaneously viewing the terrain shape around the work machine and the generated trajectory, the user can more appropriately judge the validity of the generated trajectory.

In addition, in this embodiment, the display unit may display a moving image of the working part moving along the trajectory generated by the generation unit, superimposed on an image showing the terrain shape around the work machine.

This allows users to more appropriately judge the validity of the generated trajectory by checking the video image.

In addition, in this embodiment, the display unit may display an image representing the shape of the work object around the work machine, predicted after the work part operates along the trajectory generated by the generation unit.

This allows the validity of the generated trajectory to be more appropriately determined by checking the shape of the work object after the work part moves along the generated trajectory.

In addition, in this embodiment, the assistance device may include a control unit that automatically operates the work machine based on the trajectory generated by the generation unit in response to user input. The control unit is, for example, the operation control unit 302G.

This allows even relatively inexperienced operators to operate the work part along the target trajectory, thereby improving work efficiency and user convenience.

In addition, in this embodiment, the work machine may be equipped with the above-mentioned operation assistance device.

This allows the work machine to generate a target trajectory more easily and improves work efficiency.

Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist described in the claims.

1 Lower traveling body 1C, 1CL, 1CR Crawler 1ML, 1MR Travel hydraulic motor 2 Swing mechanism 2M Swing hydraulic motor 3 Upper rotating body 4 Boom 5 Arm 6 Bucket 7 Boom cylinder 8 Arm cylinder 9 Bucket cylinder 10 Cabin 11 Engine 13 Regulator 14 Main pump 15 Pilot pump 17 Control valve 26 Operation device 29 Operation pressure sensor 30 Controller 31 Hydraulic control valve 32 Shuttle valve 33 Hydraulic control valve 40 Imaging device 50 Output device 50A Display device 52 Input device 60 Communication device 100 Excavator 150 Support device 200 Information processing device 300 Remote operation support device 301 Operation log providing unit 301A Operation log recording unit 301B Operation log storage unit 301C Operation log transmission unit 302 Work support unit 302A Learned model storage unit 302B Work object shape acquisition unit 302C Operation selection unit 302D, condition setting unit 302E, trajectory generation unit 302F, display processing unit 302G, operation control units 700, 800, 900, screen 2001, operation log acquisition unit 2002, operation log storage unit 2003, teacher data generation unit 2004, machine learning unit 2005, trained model storage unit 2006, distribution unit AT, attachment HA, hydraulic actuator LM, trained models S1 to S5, sensor SYS, operation support system

Claims (11)

an acquisition unit that acquires data relating to the shape of a work target in the vicinity of the work machine;
a display unit that displays an image representing the shape of the work object based on the data acquired by the acquisition unit;
a setting unit that sets a point on the shape of the work object that will be a target when the work machine is operating;
a generation unit that generates a trajectory of a working portion of the work machine based on the data acquired by the acquisition unit, the target shape of the work object, and the points set by the setting unit,
Support equipment. a selection unit that selects one operation from a plurality of candidate operations of the work machine in response to an input from a user;
the generation unit generates a trajectory of the working portion resulting from the one operation of the work machine based on the data acquired by the acquisition unit and the points set by the setting unit.
The support device according to claim 1 . the setting unit sets a target point on the topographical shape around the work machine during operation of the work machine and an attitude of the work part corresponding to that point in response to input from a user.
The support device according to claim 1 or 2. the display unit displays an image representing the trajectory generated by the generation unit, superimposed on an image representing the shape of the terrain around the work machine.
4. The assistance device according to claim 1. the display unit displays a moving image of the working part moving along the trajectory generated by the generation unit, superimposed on an image representing the topographical shape around the work machine.
The support device according to claim 4. the display unit displays an image representing a shape of the work object around the work machine that is predicted after the work part moves along the trajectory generated by the generation unit.
6. The support device according to claim 4 or 5. a control unit that automatically operates the work machine based on the trajectory generated by the generation unit in response to a user input;
7. An assistance device according to any one of claims 1 to 6. the setting unit sets a target point on the shape of the work object during operation of the work machine based on the data acquired by the acquisition unit.
8. An assistance device according to any one of claims 1 to 7. the generation unit uses a trained model that has been machine-learned using teacher data related to the operation of the work machine as operated by a relatively skilled operator, which is associated with the shape of the work object, to generate a trajectory of the working portion of the work machine based on the data acquired by the acquisition unit, the target shape of the work object, and the points set by the setting unit.
9. An assistance device according to any one of claims 1 to 8. A support device according to any one of claims 1 to 9 ,
Work machinery. Support devices include:
an acquisition step of acquiring data relating to the shape of a work object around the work machine;
a display step of displaying an image representing the shape of the work object on a display unit based on the data acquired in the acquisition step;
a setting step of setting a point on the shape of the work object that will be a target when the work machine is operating;
a generating step of generating a trajectory of a working portion of the work machine based on the data acquired in the acquiring step, the target shape of the work object, and the points set in the setting step;
program.