JP7616511B2 - Remote operation support system, remote operation support device - Google Patents

Description

This disclosure relates to a remote operation support system for a work machine, etc.

Technology relating to remote control of work machines is known in the past (see, for example, Patent Document 1).

In Patent Document 1, an imaging device and a distance sensor are installed on a work machine (shovel), and when a monitored object such as a person (hereinafter, "monitored object") is detected within a specified range around the work machine from the output of these, an alert is issued to the operator of an external device that remotely controls the work machine.

International Publication No. 2020/218455

However, communication delays can occur between the work machine and the external device. Therefore, if the communication delay is relatively large, the time lag between when an image capture device, distance sensor, etc. detects that a person, etc., to be monitored has entered a specified range and when a notification is actually given to an operator, etc., of the external device that remotely operates the work machine, can become relatively large. As a result, when a notification is given to an operator, etc., of the external device, the distance between the person, etc., to be monitored and the work machine can become even closer. Therefore, there is room for improvement in terms of excavator safety.

In view of the above issues, the objective is to provide technology that can further improve the safety of work machines when remotely operated.

In order to achieve the above object, in one embodiment of the present disclosure,
An acquisition device provided on a work machine for acquiring data relating to objects in the vicinity of the work machine;
a work machine communication device that is provided in the work machine and communicates with an external device that assists in remote operation of the work machine;
an external device communication device provided in the external device and configured to communicate with the work machine;
a remote operation device provided in the external device and configured to receive an input for remotely operating the work machine;
an object detection device that is provided on the work machine or the external device and detects a predetermined object around the work machine based on an output of the acquisition device;
a notification device that is provided in the external device and that, when the predetermined object is detected by the object detection device within a predetermined range based on the work machine, notifies a user of the remote control device of that fact,
when a communication delay between the work machine and the external device is relatively large, the outer edge of the predetermined range is changed in a direction away from the work machine compared to when the communication delay is relatively small.
A remote operation support system is provided.

In another embodiment of the present disclosure,
a communication device that communicates with a work machine having an acquisition device that acquires data related to objects around the work machine;
a remote control device that receives an input for remotely controlling the work machine;
a notification device that notifies a user of the remote control device when a predetermined object is detected within a predetermined range based on the work machine based on the output of the acquisition device,
when a communication delay between the work machine and the work machine is relatively large, the outer edge of the predetermined range is changed in a direction away from the work machine compared to when the communication delay is relatively small.
A remote operation support device is provided.

The above-described embodiment can further improve the safety of the work machine when remote operation is performed.

FIG. 1 is a schematic diagram illustrating an example of a remote operation support system. FIG. 2 is a side view showing an example of a shovel. FIG. 2 is a top view showing an example of a shovel. FIG. 1 is a diagram illustrating an example of a crawler crane. FIG. 1 illustrates an example of a configuration of a remote operation support system. FIG. 1 is a diagram illustrating an example of a configuration of a shovel. FIG. 1 is a diagram illustrating an example of a configuration of a crawler crane. 6 is a flowchart illustrating a first example of a process for setting a notification range by a control device. 10 is a flowchart illustrating a second example of a process for setting a notification range by a control device. 10 is a flowchart illustrating a third example of a process for setting a notification range by a control device. FIG. 13 is a diagram showing an example of an image of the shovel's surroundings displayed on the display device. FIG. 2 is a diagram showing an example of an image of the surroundings of a crawler crane displayed on a display device. FIG. 11 is a diagram showing another example of the image of the shovel's surroundings displayed on the display device. FIG. 13 is a diagram showing another example of the image of the vicinity of the crawler crane displayed on the display device.

The following describes the embodiment with reference to the drawings.

[Outline of remote operation support system]
First, an overview of a remote operation support system SYS according to this embodiment will be described with reference to FIGS.

Figure 1 is a schematic diagram showing an example of a remote operation support system SYS according to this embodiment. Figures 2 to 4 are diagrams showing a specific example of a work machine 100 included in the remote operation support system SYS. Specifically, Figures 2 and 3 are a side view and a top view showing an example of an excavator 100A, and Figure 4 is a diagram showing an example of a crawler crane 100B. In the following description, corresponding components of the excavator 100A and the crawler crane 100B (e.g., components having the same function) are denoted by the same reference numerals.

As shown in FIG. 1, the remote operation support system SYS includes a work machine 100 and a remote operation support device 200.

The remote operation support system SYS may include one or more work machines 100. Furthermore, when the remote operation support system SYS includes multiple work machines 100, the multiple work machines 100 may all be of the same type, or at least some of the work machines 100 may be of different types. For example, the remote operation support system SYS may include multiple units of either the shovel 100A or the crawler crane 100B described below, or may include both the shovel 100A and the crawler crane 100B.

The remote operation support system SYS may include multiple remote operation support devices 200. That is, the multiple remote operation support devices 200 may perform processing related to the remote operation support system SYS in a distributed manner. For example, each of the multiple remote operation support devices 200 may communicate with a portion of the work machines 100 that it is responsible for among all the work machines 100 included in the remote operation support system SYS, and perform processing targeted at the portion of the work machines 100.

The remote operation support system SYS supports the remote operation of the work machine 100 using the remote operation support device 200.

Furthermore, as described below, when the work machine 100 performs work by fully automatic driving, the remote operation support system SYS may, for example, support the remote monitoring of the work performed by the fully automatic driving of the work machine 100 in the remote operation support device 200. In this case, the remote operation support system SYS may support the operation of the work machine 100 by the intervention of a monitor for the fully automatic driving function in the remote operation support device 200.

<Overview of the work machine>
2 to 4, the work machine 100 includes a lower traveling body 1, an upper rotating body 3 rotatably mounted on the lower traveling body 1 via a rotating mechanism 2, an attachment AT attached to the upper rotating body 3, and a cabin 10 for an operator to ride in. Hereinafter, the front of the work machine 100 (upper rotating body 3) corresponds to the direction in which the attachment for the upper rotating body 3 extends when the excavator 100A is viewed in a plan view (top view) from directly above along the rotation axis of the upper rotating body 3.

The work machine 100 may be a machine other than the excavator 100A or the crawler crane 100B. For example, the work machine 100 may be a forklift, a lifting magnet machine with a lifting magnet attached as an end attachment, a bulldozer, a wheel loader, an asphalt finisher, a forestry machine, etc.

The work machine 100 can perform various tasks by operating at least one of the lower traveling body 1, the upper rotating body 3, and the attachment AT.

The work machine 100 is also equipped with a communication device 60, and can communicate with the remote operation support device 200 via a specified communication line NW. This allows the work machine 100 to transmit (upload) various information to the remote operation support device 200 and receive various signals (e.g., information signals and control signals) from the remote operation support device 200.

The communication line NW may include, for example, a wide area network (WAN). The wide area network may include, for example, a mobile communication network with a base station as its terminal. The wide area network may also include, for example, a satellite communication network that uses a communication satellite above the work machine 100. The wide area network may also include, for example, the Internet network. The communication line NW may also include, for example, a local network (LAN) of a facility in which the remote operation support device 200 is installed. The local network may be a wireless line, a wired line, or a line that includes both. The communication line NW may also include, for example, a short-distance communication line based on a predetermined wireless communication method such as WiFi or Bluetooth (registered trademark).

The work machine 100 is configured to be remotely operated from the remote operation support device 200. When the work machine 100 is remotely operated, the inside of the cabin 10 may be unmanned.

Specifically, the work machine 100 is operated by input from a user (operator) regarding the actuator of the work machine 100 performed by the remote operation support device 200. In this case, the work machine 100 transmits data of images (hereinafter, "peripheral images") showing the surroundings of the work machine 100, such as images captured by the imaging devices 40 and 42 described below and processed images (e.g., viewpoint conversion images) based on the captured images, to an external device. The peripheral images are then displayed on a display device 230A described below provided in the remote operation support device 200. In addition, various information images (information screens) displayed on the output device 50 (display device) in the cabin 10 of the work machine 100 may also be displayed on the display device 230A of the remote operation support device 200. This allows the operator to remotely operate the work machine 100 while checking the display contents, such as the peripheral images showing the surroundings of the work machine 100 and various information images displayed on the display device 230A. The work machine 100 may then operate actuators in response to a remote operation signal indicating the content of the remote operation received from the remote operation support device 200, and drive driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT.

The work machine 100 may also operate actuators (e.g., hydraulic actuators) in response to operation by an operator in the cabin 10 to drive driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT. In the following, the explanation will be given on the assumption that the operation by the operator includes both operation of the operating device 26 by the operator in the cabin 10 and remote operation by an external operator.

The work machine 100 may also automatically operate the actuators regardless of the content of the operator's operation. This allows the work machine 100 to realize a function for automatically operating at least some of the driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT, i.e., a so-called "automatic driving function" or "machine control (MC) function."

The automatic driving function may include a function to automatically operate a driven element (actuator) other than the driven element (actuator) to be operated in response to the operation of the operator, i.e., a so-called "semi-automatic driving function" or "operation support type MC function". The automatic driving function may also include a function to automatically operate at least a part of a plurality of driven elements (hydraulic actuators) on the assumption that there is no operation by the operator, i.e., a so-called "fully automatic driving function" or "fully automatic type MC function". When the fully automatic driving function is enabled in the work machine 100, the inside of the cabin 10 may be unmanned. The semi-automatic driving function and the fully automatic driving function may also include a mode in which the operation content of the driven element (actuator) to be the target of the automatic driving is automatically determined according to a rule that is specified in advance. The semi-automatic driving function and the fully automatic driving function may also include a so-called "autonomous driving function" in which the work machine 100 autonomously makes various judgments and autonomously determines the operation content of the driven element (hydraulic actuator) to be the target of the automatic driving according to the judgment results.

The work machine 100 is, for example, a shovel 100A.

As shown in Figures 2 and 3, the excavator 100A includes a lower running body 1, an upper rotating body 3 that is mounted on the lower running body 1 so as to be freely rotatable via a rotating mechanism 2, an attachment AT for performing various tasks, and a cabin 10.

The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C. The lower traveling body 1 causes the excavator 100A to travel by hydraulically driving each crawler 1C with a left traveling hydraulic motor 1ML and a right traveling hydraulic motor 1MR.

The upper rotating body 3 rotates relative to the lower traveling body 1 as the rotating mechanism 2 is hydraulically driven by the rotating hydraulic motor 2M.

The attachment AT includes a boom 4A, an arm 5A, and a bucket 6A as driven elements.

The boom 4A is attached to the center of the front of the upper rotating body 3 so that it can be raised and lowered, an arm 5A is attached to the tip of the boom 4A so that it can rotate up and down, and a bucket 6A is attached to the tip of the arm 5A so that it can rotate up and down.

The bucket 6A is an example of an end attachment. The bucket 6A is used, for example, for excavation work. In addition, instead of the bucket 6A, another end attachment may be attached to the tip of the arm 5A depending on the work content, etc. The other end attachment may be, for example, another type of bucket, such as a large bucket, a slope bucket, or a dredging bucket. The other end attachment may be a type of end attachment other than a bucket, such as an agitator, a breaker, or a grapple. In addition, a spare attachment, such as a quick coupling or a tiltrotator, may be interposed between the arm 5A and the end attachment.

A hook HK for crane work may also be attached to the bucket 6A. The base end of the hook HK is rotatably connected to a bucket pin that connects the arm 5A and the bucket 6A. As a result, the hook HK is stored in the space formed between the two bucket links when work other than crane work (hoisting work), such as excavation work, is being performed.

The boom 4A, arm 5A, and bucket 6A are hydraulically driven by hydraulic actuators, a boom cylinder 7A, an arm cylinder 8A, and a bucket cylinder 9A, respectively.

In addition, in the shovel 100A, some or all of the various hydraulic actuators may be replaced with electric actuators. In other words, the shovel 100A may be a hybrid shovel or an electric shovel.

The cabin 10 is a cockpit where the operator sits, and is mounted, for example, on the front left side of the upper rotating body 3.

Note that if an operator does not board and operate the excavator 100A, and the work machine 100 operates solely by remote control or fully automatic operation, the cabin 10 may be omitted. The same applies to the crawler crane 100B below.

The work machine 100 is, for example, a crawler crane 100B.

As shown in FIG. 4, the crawler crane 100B includes a lower running body 1, an upper rotating body 3 mounted on the lower running body 1 so as to be freely rotatable via a rotating mechanism 2, an attachment AT, a mast 5B, a backstop 6B, a hook HK, a counterweight 9B, and a cabin 10.

The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C. The lower traveling body 1 drives the crawler crane 100B by hydraulically driving each crawler 1C with a left traveling hydraulic motor 1ML and a right traveling hydraulic motor 1MR (see FIG. 7).

The upper rotating body 3 rotates relative to the lower traveling body 1 as the rotating mechanism 2 is hydraulically driven by the rotating hydraulic motor 2M (see Figure 7).

The attachment AT (an example of a working device) includes a boom 4B and a main hoisting rope 7B.

The boom 4B is movably attached to the center of the front of the upper rotating body 3. A main hoisting rope 7B hangs down from the tip of the boom 4B, and a hook HK is attached to the tip of the main hoisting rope 7B. That is, the hook HK is attached to the tip of the boom 4B via the main hoisting rope 7B.

The main hoisting rope 7B has its base end attached to a main hoisting winch 7Ba that is attached to the rear portion between the base end and tip of the boom 4B, and its tip is attached to the hook HK. The main hoisting rope 7B can be wound up and unwound by the main hoisting winch 7Ba, which is hydraulically driven by a main hoisting hydraulic motor 7M (see Figure 7), to raise and lower the hook HK.

The mast 5B is attached to the upper rotating body 3 slightly behind the base end of the boom 4B and is rotatable around a rotation axis parallel to the rotation axis of the boom 4B. The tip of the mast 5B is connected to the tip of the boom 4B via a pendant rope 5Ba, and the boom hoisting winch 5Bc, which is hydraulically driven by the hoisting hydraulic motor 5M (see Figure 7), winds and unwinds the boom hoisting rope 5Bb, causing the boom hoisting boom 4B to rise and fall via the mast 5B.

As with the excavator 100A, the crawler crane 100B may have some or all of the hydraulic actuators replaced with electric actuators.

The backstop 6B is attached so that its base end can rotate about a rotation axis parallel to the rotation axis of the boom 4B in a portion of the upper rotating body 3 rearward of the base end of the boom 4B, and its tip is attached so that it can rotate about a rotation axis parallel to the rotation axis of the boom 4B in a rear portion between the base end and tip of the boom 4B. The backstop 6B expands and contracts in response to the raising and lowering operation of the boom 4B, and has the function of supporting the boom 4B from the rear when the boom 4B is in a substantially upright state, for example.

The hook HK is attached to the end of the main hoisting rope 7B and is used to suspend a load.

The counterweight 9B is provided at the rear end of the upper rotating body 3 and serves to balance the weight of the boom 4B and the suspended load SL.

The cabin 10 is attached, for example, to the right front end of the upper rotating body 3. Inside the cabin 10, a cockpit and an operating device 26 (see Figure 7) for operating various actuators are provided.

<Outline of remote operation support device>
The remote operation support device 200 (an example of an external device) supports the remote operation of the work machine 100. The remote operation support device 200 may also support the remote monitoring of the work machine 100 that performs work in a fully automated manner.

The remote operation support device 200 may be, for example, an on-premise server or a cloud server installed in a management center outside the work site where the work machine 100 performs work. The remote operation support device 200 may also be, for example, an edge server installed in the work site where the work machine 100 performs work, or in a location relatively close to the work site. The remote operation support device 200 may also be a stationary terminal device or a portable terminal device (mobile terminal) installed in a management office or the like in the work site of the work machine 100. The stationary terminal device may include, for example, a desktop computer terminal. The portable terminal device may include, for example, a smartphone, a tablet terminal, a laptop computer terminal, etc.

The remote operation support device 200 has, for example, a communication device 220 (see Figures 5 to 7), and as described above, communicates with the work machine 100 through the communication line NW. This allows the remote operation support device 200 to receive various information uploaded from the work machine 100 and transmit various signals to the work machine 100. Therefore, the user of the remote operation support device 200 can check various information related to the work machine 100 through the output device 230 (see Figures 2 and 3). In addition, the remote operation support device 200 can, for example, transmit information signals to the work machine 100 to provide information necessary for work, or transmit control signals to control the work machine 100. The users of the remote operation support device 200 may include, for example, the owner of the work machine 100, the manager of the work machine 100, an engineer of the manufacturer of the work machine 100, the operator of the work machine 100, the manager, supervisor, worker, etc. of the work site of the work machine 100.

For example, the remote operation support device 200 has a remote operation device 240A through which the operator inputs information for remote operation, and a display device 230A that displays an image (peripheral image) of the surroundings of the work machine 100. A signal input from the remote operation device 240A is transmitted from the remote operation support device 200 to the work machine 100 as a remote operation signal. This allows the user (operator) of the remote operation support device 200 to remotely operate the work machine 100 using the remote operation device 240A while checking the surroundings of the work machine 100 on the display device 230A.

For example, a user (monitor) of the remote operation assistance device 200 can monitor the state of work of the work machine 100 on the display device 230A. Also, for example, when an abnormality occurs in the work machine 100 or when the operation of the work machine 100 is inappropriate, the user (monitor) of the remote operation assistance device 200 can use the remote operation device 240A to perform an intervention operation on the automatic driving function of the work machine 100. Therefore, the monitor can use the remote operation device 240A to perform an emergency stop of the work machine 100 or to perform remote operation to make the work machine 100 perform an appropriate operation.

[Configuration of remote operation support system]
Next, the configuration of the remote operation support system SYS will be described with reference to FIGS.

Fig. 5 is a block diagram showing an example of the configuration of the remote operation support system SYS according to this embodiment. Fig. 6 is a diagram showing an example of the configuration of a shovel 100A. Specifically, Fig. 6 is a diagram showing a specific example of the configuration of a shovel 100A corresponding to Fig. 5. Fig. 7 is a diagram showing an example of the configuration of a crawler crane 100B. Specifically, Fig. 7 is a diagram showing a specific example of the configuration of a crawler crane 100B corresponding to Fig. 5.

In Figures 5 to 7, the paths through which mechanical power is transmitted are indicated by double lines, the paths through which high-pressure hydraulic oil that drives the hydraulic actuator flows are indicated by solid lines, the paths through which pilot pressure is transmitted are indicated by dashed lines, and the paths through which electrical signals are transmitted are indicated by dotted lines.

<Configuration of the work machine>
The work machine 100 includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside, and a control system for various controls.

<<Hydraulic drive system>>
5, the hydraulic drive system of the work machine 100 according to this embodiment includes hydraulic actuators HA that hydraulically drive each of the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, and the attachment AT, as described above. The hydraulic drive system of the work machine 100 according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

As shown in FIG. 6, when the work machine 100 is a shovel 100A, the hydraulic actuator HA includes a traveling hydraulic motor 1ML, 1MR, a swing hydraulic motor 2M, a boom cylinder 7A, an arm cylinder 8A, and a bucket cylinder 9A.

Also, as shown in FIG. 7, when the work machine 100 is a crawler crane 100B, the hydraulic actuator HA includes a traveling hydraulic motor 1ML, 1MR, a swing hydraulic motor 2M, a hoisting hydraulic motor 5M, and a main hoisting hydraulic motor 7M, etc.

The engine 11 is the prime mover of the work machine 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses diesel as fuel. The engine 11 is mounted, for example, at the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by the controller 30 (described later), and drives the main pump 14 and the pilot pump 15.

In addition, other prime movers (e.g., electric motors) may be installed in the work machine 100 instead of or in addition to the engine 11.

The regulator 13 controls (adjusts) the discharge volume of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter, the "tilt angle") in response to a control command from the controller 30.

The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, at the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13 to adjust the stroke length of the piston and control the discharge flow rate (discharge pressure).

The control valve 17 is a hydraulic control device that controls the hydraulic actuator HA in response to the contents of the operator's operation of the operating device 26 or remote operation, or in response to an operation command related to the automatic operation function output from the controller 30. The operation command corresponding to the automatic operation function may be generated by the controller 30, or may be generated by another control device (computing device) that controls the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to each hydraulic actuator in response to the operator's operation or an operation command corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (also called "directional control valves") that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

<<Operation system>>
As shown in FIG. 5 , the operating system of the work machine 100 according to this embodiment includes the pilot pump 15 , an operating device 26 , a hydraulic control valve 31 , a shuttle valve 32 , and a hydraulic control valve 33 .

The pilot pump 15 supplies pilot pressure to various hydraulic equipment via a pilot line 25. The pilot pump 15 is mounted, for example, at the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

The pilot pump 15 may be omitted. In this case, the relatively high pressure hydraulic oil discharged from the main pump 14 is reduced in pressure by a specified pressure reducing valve, and the relatively low pressure hydraulic oil is supplied to various hydraulic equipment as pilot pressure.

The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate the various driven elements. In other words, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive each driven element. The operating device 26 includes a pedal device and a lever device for operating each driven element (hydraulic actuator HA).

For example, as shown in FIG. 4, the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and the pilot line 25A branched from the pilot line 25, and outputs pilot pressure corresponding to the operation to the secondary pilot line 27A. The pilot line 27A is connected to one inlet port of the shuttle valve 32, and is connected to the control valve 17 via the pilot line 27 connected to the outlet port of the shuttle valve 32. As a result, pilot pressure corresponding to the operation of various driven elements (hydraulic actuators) in the operating device 26 can be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by the operator, etc.

The operating device 26 may be an electric type. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter, "operation signal") according to the operation content, and the operation signal is taken in by the controller 30. The controller 30 then outputs a control command according to the operation signal, that is, a control signal according to the operation content for the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure according to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26. In addition, the control valve (directional control valve) built into the control valve 17 for driving each hydraulic actuator may be an electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17, that is, to the electromagnetic solenoid type control valve. Furthermore, if the work machine 100 is remotely operated or operates using an automatic driving function, the operating device 26 may be omitted.

The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26. The hydraulic control valve 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and may be configured to change its flow path area (i.e., the cross-sectional area through which hydraulic oil can flow). As a result, the hydraulic control valve 31 can output a predetermined pilot pressure to the secondary pilot line 27B using the hydraulic oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG. 4, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure corresponding to a control signal from the controller 30 to the control valve 17 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. Also, as shown in FIG. 5, the hydraulic control valve 31 can directly apply a predetermined pilot pressure corresponding to a control signal from the controller 30 to the control valve 17 through the pilot line 27B and the pilot line 27. Therefore, the controller 30 can supply pilot pressure from the hydraulic control valve 31 to the control valve 17 according to the operation of the electric operating device 26, thereby realizing the operation of the work machine 100 based on the operation of the operator.

The controller 30 may also control, for example, the hydraulic control valve 31 to realize the automatic driving function. Specifically, the controller 30 outputs a control signal corresponding to an operation command related to the automatic driving function to the hydraulic control valve 31, regardless of whether the operating device 26 is operated or not. As a result, the controller 30 can supply pilot pressure corresponding to the operation command related to the automatic driving function from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the work machine 100 based on the automatic driving function.

The controller 30 may also control, for example, the hydraulic control valve 31 to realize remote operation of the work machine 100. Specifically, the controller 30 outputs a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the remote operation assistance device 200 to the hydraulic control valve 31 via the communication device 60. As a result, the controller 30 can supply pilot pressure corresponding to the content of the remote operation from the hydraulic control valve 31 to the control valve 17, thereby realizing the operation of the work machine 100 based on the remote operation of the operator.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic oil having the higher pilot pressure of the two pilot pressures input to the two inlet ports to the outlet port. The shuttle valve 32 is provided for each driven element (hydraulic actuator HA) that is the object of operation of the operating device 26. One of the two inlet ports of the shuttle valve 32 is connected to the secondary pilot line 27A of the operating device 26 (specifically, the above-mentioned lever device or pedal device included in the operating device 26), and the other is connected to the secondary pilot line 27B of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 through the pilot line 27. The corresponding control valve is a control valve that drives the hydraulic actuator that is the object of operation of the above-mentioned lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure of the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure of the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, the controller 30 can control the corresponding control valve regardless of the operator's operation of the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper rotating body 3, attachment AT) regardless of the operating state of the operating device 26 by the operator, thereby realizing an automatic driving function.

The hydraulic control valve 33 is provided in the pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured to be able to change its flow area, for example. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. As a result, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by the operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. In addition, the controller 30 can reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31, for example, even when the operating device 26 is being operated. Therefore, the controller 30 can reliably apply the desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation content of the operating device 26, by controlling the hydraulic control valve 31 and the hydraulic control valve 33. Therefore, for example, the controller 30 can more appropriately realize the automatic driving function and remote control function of the work machine 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31.

<<User Interface>>
As shown in FIG. 5 , the user interface system of the work machine 100 according to this embodiment includes the operation device 26 , an output device 50 , and an input device 52 .

The output device 50 outputs various information to the user of the work machine 100 (e.g., the operator of the cabin 10 or a work vehicle near the work machine 100).

For example, the output device 50 includes lighting equipment and display devices that output various information in a visual manner. The lighting equipment is, for example, a warning light. The display device is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. The lighting equipment and display device may be provided, for example, inside the cabin 10, and output various information in a visual manner to an operator or the like inside the cabin 10. The lighting equipment and display device may also be provided, for example, on the side of the upper rotating body 3, and output various information in a visual manner to workers or the like around the work machine 100.

In addition, for example, the output device 50 includes a sound output device that outputs various information by an auditory method. Sound output devices include, for example, a buzzer and a speaker. The sound output device may be provided, for example, inside the cabin 10, and may output various information by an auditory method to an operator or the like inside the cabin 10.

Also, for example, the output device 50 may include a device that outputs various information in a tactile manner, such as by vibration of the cockpit.

The input device 52 accepts various inputs from the user of the work machine 100, and signals corresponding to the accepted inputs are taken into the controller 30. The input device 52 is provided, for example, inside the cabin 10, and accepts inputs from an operator or the like inside the cabin 10. The input device 52 may also be provided, for example, on the side of the upper rotating body 3, and accept inputs from workers or the like in the vicinity of the work machine 100.

For example, the input device 52 is an operation input device that accepts operation input. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), etc.

Also, for example, the input device 52 may be an audio input device that accepts audio input from the user. The audio input device includes, for example, a microphone.

For example, the input device 52 may be a gesture input device that accepts gesture input from a user. The gesture input device includes, for example, an imaging device that captures an image of a gesture made by the user.

<<Communications>>
As shown in FIG. 5 , the communication system of the work machine 100 according to this embodiment includes a communication device 60 .

The communication device 60 (an example of a communication device for a work device) is connected to the communication line NW and communicates with a device (for example, the remote operation support device 200) provided separately from the work machine 100. The device provided separately from the work machine 100 may include a device external to the work machine 100, as well as a portable terminal device brought into the cabin 10 by the user of the work machine 100. The communication device 60 may include, for example, a mobile communication module that complies with standards such as 4G ( ^4th Generation) and 5G ( ^5th Generation). The communication device 60 may also include, for example, a satellite communication module. The communication device 60 may also include, for example, a WiFi communication module or a Bluetooth (registered trademark) communication module.

<<Control system>>
5, the control system of the work machine 100 according to this embodiment includes a controller 30. The control system of the work machine 100 according to this embodiment also includes an operating pressure sensor 29, image capture devices 40, 42, and an acquisition device SX.

The controller 30 (an example of an object detection device) performs various controls related to the work machine 100. The functions of the controller 30 may be realized by any hardware or any combination of hardware and software. For example, the controller 30 is configured around a computer including a CPU (Central Processing Unit), a memory device such as a RAM (Random Access Memory), a non-volatile auxiliary storage device such as a ROM (Read Only Memory), and various interface devices for input and output.

The controller 30 includes, as functional units, an operation control unit 301, an image transmission unit 302, an object detection unit 303, and a safety control unit 304. The functions of the operation control unit 301, the image transmission unit 302, the object detection unit 303, and the safety control unit 304 are realized, for example, by loading a program installed in an auxiliary storage device into a memory device and executing the program on a CPU.

Note that some of the functions of the controller 30 may be realized by another controller (control device). In other words, the functions of the controller 30 may be realized in a distributed manner by multiple controllers.

The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, i.e., the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure by the operating pressure sensor 29 corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 is input to the controller 30.

If the operating device 26 is an electrical type, the operating pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal received from the operating device 26.

The imaging device 40 (an example of an acquisition device) acquires images to fill in blind spots and areas that are difficult to see from the operator's perspective around the excavator 100A. The output of the imaging device 40 is input to the controller 30.

The imaging device 40 may include, for example, a monocular camera, a stereo camera, a depth camera, etc. Furthermore, the imaging device 40 may acquire three-dimensional data (for example, point cloud data or surface data) representing the positions and shapes of objects around the work machine 100 within a predetermined imaging range (angle of view) based on the captured image.

2 and 3, the imaging device 40 of the excavator 100A includes a camera 40B that images the rear of the upper rotating body 3, a camera 40L that images the left side of the upper rotating body 3, and a camera 40R that images the right side of the upper rotating body 3. This allows the operator to view the images captured by the imaging device 40 and peripheral images such as processing images generated based on the captured images through the output device 50 and the display device 230A, and check the conditions to the left, right, and rear of the upper rotating body 3.

The dashed double-dashed lines in Figure 3 represent the top view angle (imaging range) of cameras 40B, 40L, and 40R.

Also, for example, as shown in FIG. 4, the imaging device 40 of the crawler crane 100B is attached to the boom 4B, for example, and acquires an image of the surroundings of the hook HK and the load suspended by the hook HK as viewed from above. This allows the operator to view the captured image of the imaging device 40 and surrounding images such as processed images generated based on the captured image through the output device 50 (display device) and the display device 230A, and to check the surroundings of the load suspended by the hook HK from an overhead perspective.

The excavator 100A may be provided with an imaging device 40 in the form shown in FIG. 4, and the crawler crane 100B may be provided with an imaging device 40 (cameras 40B, 40L, 40R) in the form shown in FIGS. 2 and 3.

The imaging device 42 captures images of the surroundings of the work machine 100 so that a remote operator or a remote monitor can check the state of work being performed by the attachment AT. The output of the imaging device 42 is input to the controller 30.

The imaging device 42 may include, for example, a monocular camera, a stereo camera, a depth camera, etc. The imaging device 42 may also acquire three-dimensional data (for example, point cloud data or surface data) representing the positions and shapes of objects around the work machine 100 within a predetermined imaging range (angle of view) based on the captured image.

2 and 3, the imaging device 42 of the shovel 100A is provided at the front of the upper rotating body 3, for example, at the front top surface of the cabin 10, and captures an image of the front of the upper rotating body 3 including the tip of the attachment AT (bucket 6A). This allows the operator to operate the shovel 100A while checking the operation of the attachment AT including the bucket 6A by visually checking the image captured by the imaging device 42 and peripheral images such as processing images generated based on the imaging device 42 through the display device 230A.

Also, for example, as shown in FIG. 4, the imaging device 42 of the crawler crane 100B is provided at the front of the upper rotating body 3, such as the front part of the upper surface of the cabin 10, and captures the state in front of the upper rotating body 3 including the hook HK. Specifically, the imaging device 42 includes a camera 42A that captures an image of a relatively high vertical space away from the ground diagonally above and in front of the upper rotating body 3, and a camera 42B that captures an image of a relatively low vertical range near the ground diagonally below and in front of the upper rotating body 3. This allows the operator to operate the crawler crane 100B while checking the movement of the hook HK by visually checking the peripheral images, such as the image captured by the imaging device 42 and the processed image generated based on the captured image, through the display device 230A.

The acquisition device SX acquires information about the state of the shovel 100A and the state of the surroundings of the shovel 100A. The output of the acquisition device SX is input to the controller 30.

As shown in FIG. 6, when the work machine 100 is a shovel 100A, the acquisition device SX includes a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a machine attitude sensor S4, and a swing angle sensor S5.

Also, as shown in FIG. 7, when the work machine 100 is a crawler crane 100B, the acquisition device SX includes a machine attitude sensor S4 and a rotation angle sensor S5.

The boom angle sensor S1 acquires detection information regarding the attitude angle of the boom 4A (hereinafter, "boom angle") relative to a predetermined reference (e.g., a horizontal plane or one of the states at either end of the movable angle range of the boom 4A). The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. The boom angle sensor S1 may also include a cylinder sensor capable of detecting the extension/retraction position of the boom cylinder 7A.

The arm angle sensor S2 acquires detection information regarding the posture angle of the arm 5A (hereinafter, "arm angle") relative to a predetermined reference (e.g., a straight line connecting the connection points at both ends of the boom 4A or one of the states at both ends of the movable angle range of the arm 5A). The arm angle sensor S2 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU, etc. The arm angle sensor S2 may also include a cylinder sensor capable of detecting the extension/retraction position of the arm cylinder 8A.

The bucket angle sensor S3 acquires detection information regarding the attitude angle of the bucket 6A (hereinafter, "bucket angle") relative to a predetermined reference (e.g., a straight line connecting the connection points at both ends of the arm 5A or a state at either end of the movable angle range of the bucket 6A). The bucket angle sensor S3 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU, etc. The bucket angle sensor S3 may also include a cylinder sensor capable of detecting the extension/retraction position of the bucket cylinder 9A.

The machine attitude sensor S4 acquires detection information related to the attitude state of the machine of the work machine 100, which includes the undercarriage 1 and the upper rotating body 3. The machine attitude sensor S4 is mounted, for example, on the upper rotating body 3, and acquires detection information related to the inclination angle of the upper rotating body 3 with respect to the horizontal plane and the attitude angle around the rotation axis (i.e., the orientation of the upper rotating body 3 with respect to the ground). The machine attitude sensor S4 may include, for example, an acceleration sensor (tilt sensor), an angular velocity sensor, a six-axis sensor, an IMU, etc.

The rotation angle sensor S5 acquires detection information regarding the rotation angle of the upper rotating body 3 relative to the lower traveling body 1 (i.e., the orientation of the upper rotating body 3). The rotation angle sensor S5 includes, for example, a potentiometer, a rotary encoder, a resolver, etc.

Furthermore, for example, the acquisition device SX may include a positioning device capable of measuring the absolute position of the own machine. The positioning device is, for example, a Global Navigation Satellite System (GNSS) sensor. This can improve the estimation accuracy of the attitude state of the work machine 100.

For example, the acquisition device SX may also include a distance sensor (an example of an acquisition device) that detects the distance to objects in the vicinity of the work machine 100. Distance sensors include, for example, LIDAR (Light Detecting and Ranging), millimeter wave radar, ultrasonic sensors, infrared sensors, distance image sensors, etc. This allows the controller 30 (object detection unit 303) to detect objects in the vicinity of the work machine 100 using the output of the distance sensor instead of or in addition to the output of the imaging devices 40, 42.

Note that sensors S1 to S5 of shovel 100A and sensors S4 and S5 of crawler crane 100B may be omitted. For example, information about the surroundings of shovel 100A acquired by imaging devices 40 and 42 or distance sensors described below may include information about the positions and shapes of surrounding objects and attachments as seen from the machine body (upper rotating body 3). In this case, for example, controller 30 may be able to estimate the attitude state of attachment AT and the machine body (upper rotating body 3) from that information, depending on the required accuracy.

The operation control unit 301 controls the operation of the hydraulic actuator of the work machine 100, with the hydraulic control valve 31 as the control object.

The operation control unit 301 controls the operation of the hydraulic actuators HA (driven elements) of the work machine 100 based on the operation of the electric operating device 26, for example, with the hydraulic control valves 31 as the control target. Specifically, the operation control unit 301 may selectively control the hydraulic control valves 31 corresponding to each of the multiple hydraulic actuators HA (driven elements) according to the content of the operation signal output from the operating device 26.

The operation control unit 301 also performs control related to remote operation of the hydraulic actuator HA (driven element) of the work machine 100, for example, with the hydraulic control valve 31 as the control target. That is, the operation of the hydraulic actuator HA (driven element) of the work machine 100 includes remote operation of the hydraulic actuator HA from outside the work machine 100 (remote operation support device 200). Specifically, the operation control unit 301 may selectively control the hydraulic control valves 31 corresponding to each of the multiple hydraulic actuators HA (driven elements) according to the content of the remote operation signal received from the remote operation support device 200 via the communication device 60.

The operation control unit 301 also controls the automatic driving function of the work machine 100, for example, by controlling the hydraulic control valve 31 as the control object. Specifically, the operation of the hydraulic actuator HA of the work machine 100 may include an operation command for the hydraulic actuator HA of the work machine 100 that is output based on the automatic driving function. Specifically, the operation control unit 301 may selectively control the hydraulic control valve 31 corresponding to each of the multiple hydraulic actuators HA (driven elements) according to the content of the operation command.

The image transmission unit 302 transmits the captured image of the imaging device 40 or peripheral images such as processed images generated based on the captured image of the imaging device 40 to the remote operation support device 200 via the communication device 60.

For example, the image transmission unit 302 of the excavator 100A may generate a processed image (composite image) by combining images captured by multiple cameras (at least two of cameras 40B, 40L, and 40R) based on the image captured by the imaging device 40, and transmit the processed image to the remote operation support device 200.

Specifically, the image transmission unit 302 may perform known viewpoint conversion processing and synthesis processing, etc., based on the images captured by the cameras 40B, 40L, and 40R to generate a synthetic image (viewpoint conversion image) viewed from a virtual viewpoint, and transmit the synthetic image to the remote operation support device 200.

For example, the image transmission unit 302 of the crawler crane 100B may generate a viewpoint conversion image based on the image captured by the imaging device 40 and transmit it to the remote operation support device 200.

Similarly, the image transmission unit 302 transmits the captured image of the imaging device 42 or a peripheral image such as a processed image generated by the controller 30 based on the captured image of the imaging device 42 to the remote operation support device 200 via the communication device 60.

For example, the image transmission unit 302 of the shovel 100A may generate a viewpoint conversion image based on the image captured by the imaging device 42 and transmit the image to the remote operation support device 200. The image transmission unit 302 of the shovel 100A may also perform known viewpoint conversion processing and synthesis processing, etc. on the images captured by the imaging devices 40 and 42 to generate a composite image in which 360 degrees horizontally around the shovel 100A are viewed from a virtual viewpoint, and transmit the composite image to the remote operation support device 200.

The object detection unit 303 detects a specific object to be monitored (hereinafter, "monitored object") in the vicinity of the work machine 100 based on the output of the imaging device 40.

Monitored objects may include people such as workers working around the work machine 100 and work site supervisors. Monitored objects may also include any object (obstacle) other than people at the work site. Obstacles other than people at the work site may include, for example, specific terrain such as holes, ditches, and piles of dirt and sand, as well as fixed (non-moving) obstacles such as road cones, fences, utility poles, temporarily stored materials, and temporary offices at the work site. Obstacles other than people at the work site may also include, for example, movable obstacles such as other work machines and work vehicles.

The object detection unit 303 detects a monitored object in a specified monitoring area (hereinafter, for convenience, "monitoring area") around the work machine 100 (upper rotating body 3) based on the output of the imaging device 40, i.e., the image captured by the imaging device 40. The following description of the object detection unit 303 is based on the assumption that the work machine 100 is located on a horizontal plane. Specifically, the object detection unit 303 of the excavator 100A may detect a monitored object in a monitoring area extending to the rear, left, and right of the upper rotating body 3. The object detection unit 303 of the crawler crane 100B may detect a monitored object in a monitoring area around the load suspended by the hook HK.

When the above-mentioned distance sensor is mounted on the upper rotating body 3, the object detection unit 303 may detect a monitored object based on the output of the distance sensor instead of or in addition to the output of the imaging device 40. Furthermore, the object detection unit 303 may detect an object around the upper rotating body 3 based on the output of the imaging device 42 in addition to the output of the imaging device 40. That is, the object detection unit 303 may detect a monitored object in a monitoring area including the imaging range of the imaging device 42 based on the output of the imaging devices 40 and 42. Furthermore, when the work machine 100 is used exclusively for remote operation, the function of the object detection unit 303 may be transferred to the remote operation support device 200 (for example, the control device 210 (an example of an object detection device)). In this case, the output of the imaging device 40 or the distance sensor is transmitted to the remote operation support device 200 through the communication device 60.

The object detection unit 303 detects a monitored object, for example, in a horizontal direction (hereinafter simply "horizontal direction") as seen from the work machine 100, that is, in a monitoring area extending in a direction along the plane (hereinafter for convenience, "work plane") on which the work machine 100 is working (the lower running body 1 is on the ground). Specifically, the object detection unit 303 may detect a monitored object in a monitoring area in which the distance D from a specified part of the work machine 100 is within a threshold value Dth1. In this case, the threshold value Dth1 may be constant regardless of the direction as seen from the specified part of the work machine 100, or may change depending on the direction as seen from the specified part of the work machine 100. For example, the object detection unit 303 of the excavator 100A detects a monitored object in a monitoring area in which the horizontal distance D from the upper rotating body 3 is within a threshold value Dth1 (for example, 10 meters). Additionally, the object detection unit 303 of the crawler crane 100B detects a monitored object within a monitoring area whose horizontal distance from the hook HK (or the suspended load) is within a threshold value Dth1.

For example, the object detection unit 303 recognizes a surveillance object in a captured image by arbitrarily applying various known image processing methods and machine learning-based classifiers including artificial intelligence (AI). In addition, when the surveillance object is a person such as a worker, the object detection unit 303 may identify which of a plurality of pre-registered workers the recognized person corresponds to.

In addition, the object detection unit 303 may determine (estimate) the position (e.g., the position of a person's feet) (hereinafter, "actual position") of a recognized surveillance object captured in an image captured by the monocular imaging device 40 by applying various known techniques.

For example, the object detection unit 303 of the shovel 100A may estimate the horizontal position (hereinafter, "horizontal position") as seen from the upper rotating body 3 based on the size of the recognized monitored object in the captured image (e.g., the size in the height direction on the captured image). This is because there is a correlation in which the size of the recognized monitored object in the captured image becomes smaller as the monitored object moves away from the shovel 100A (upper rotating body 3). Specifically, since there is a range of expected sizes for the monitored object (e.g., the range of expected human heights), the correlation between the horizontal position of the monitored object as seen from the shovel 100A within the expected range of sizes and the size on the captured image can be specified in advance. Therefore, the object detection unit 303 can estimate the actual position (horizontal position from the upper rotating body 3) of the recognized monitored object based on a map or conversion formula that represents the correlation between the size of the monitored object in the captured image and the horizontal position as seen from the shovel 100A, which is stored in advance in an internal memory such as an auxiliary storage device of the controller 30.

Also, for example, the object detection unit 303 may estimate the actual position (e.g., foot position) of a monitored object by projective transformation (homography) of the captured image onto the plane, assuming that the monitored object exists on the same plane as the work machine 100 (specifically, the undercarriage 1). In this case, a certain portion (e.g., a certain pixel) constituting the captured image is associated with a certain position on the same plane as the work machine 100.

Furthermore, the function of the object detection unit 303 may be switched between ON (enabled) and OFF (disabled) in response to a predetermined operation by an operator or the like on the input device 52.

The safety control unit 304 controls the functional safety of the work machine 100. Specifically, the safety control unit 304 may activate a safety function when a monitored object is detected by the object detection unit 303.

The safety functions may include, for example, a notification function that outputs an alarm or the like to at least one of the inside of the cabin 10, the outside of the cabin 10, and a remote operator or manager of the work machine 100, and notifies the detection of a monitored object. This makes it possible to alert the operator inside the cabin 10, workers around the work machine 100, and operators or supervisors who remotely operate or monitor the work machine 100 that a monitored object exists within a specified range around the work machine 100. Hereinafter, the notification function to the inside of the cabin 10 (operator, etc.) is sometimes referred to as an "internal notification function", the notification function to the outside of the work machine 100 (workers, etc.) is sometimes referred to as an "external notification function", and the notification function to the operator or supervisor who remotely operates or monitors the work machine 100 is sometimes referred to as a "remote notification function".

The safety function may also include, for example, an operation restriction function that restricts the operation of the work machine 100 in response to an operation command corresponding to the operation of the operating device 26, remote operation, or the automatic driving function. This makes it possible to forcibly restrict the operation of the work machine 100 and reduce the possibility of the work machine 100 or the suspended load approaching or coming into contact with surrounding objects. The operation restriction function may also include an operation deceleration function that reduces the operating speed of the work machine 100 in response to an operation command corresponding to the operation of the operating device 26, remote operation, or the automatic driving function, slower than normal. The operation restriction function may also include an operation stop function that stops the operation of the work machine 100 and maintains it in a stopped state, regardless of the operation of the operating device 26, remote operation, or an operation command corresponding to the automatic driving function.

The safety control unit 304 activates the alarm function, for example, when the object detection unit 303 detects a monitoring target in a predetermined range (hereinafter, "alarm range") included in the monitoring area. The alarm range is set to the same range as the monitoring area, or to a range whose outer edge is relatively closer to the upper rotating body 3 or the hook HK (or the suspended load) than the monitoring area. Specifically, the safety control unit 304 may activate the alarm function when a monitored object is detected within the alarm range whose distance D from a predetermined part of the work machine 100 is within a threshold value Dth2 (≦Dth1). In this case, the threshold value Dth2 may be constant regardless of the direction as viewed from the predetermined part of the work machine 100, or may change depending on the direction as viewed from the predetermined part of the work machine 100.

The safety control unit 304, for example, controls the output device 50 (sound output device) to activate an internal alarm function or an external alarm function using sound (i.e., an auditory method) to at least one of the inside and outside of the cabin 10. At this time, the safety control unit 304 may vary the pitch, sound pressure, tone color, blowing period when blowing a sound periodically, content of the sound, etc. of the output sound according to various conditions.

The safety control unit 304 also activates an internal notification function by, for example, a visual method. Specifically, the safety control unit 304 may control the output device 50 (display device) inside the cabin 10 to display an image indicating that a monitored object has been detected on the display device. The safety control unit 304 may also highlight the monitored object shown in the peripheral image displayed on the display device inside the cabin 10 or the position on the peripheral image corresponding to the detected monitored object. More specifically, the safety control unit 304 may superimpose a frame surrounding the detected monitored object on the peripheral image displayed on the display device, or superimpose a marker on a position on the peripheral image corresponding to the actual position of the detected monitored object. This allows the output device 50 (display device) to realize a visual notification function for the operator. The safety control unit 304 may also notify the operator inside the cabin 10 that a monitored object has been detected by using a warning light, lighting device, or the like inside the cabin 10.

The safety control unit 304 may also activate an external notification function by a visual method, for example, by controlling an output device 50 (for example, a lighting device such as a headlight or a display device) provided on the side of the house part of the upper rotating body 3. The safety control unit 304 may also activate an internal notification function by a tactile method, for example, by controlling a vibration generating device that vibrates the cockpit in which the operator sits. This allows the controller 30 to make the operator, workers around the work machine 100, and supervisors aware of the presence of a monitored object (for example, a person such as a worker) around the work machine 100. Therefore, the controller 30 can prompt the operator to check the safety situation around the work machine 100, and can prompt workers in the monitored area to evacuate from the monitored area.

The safety control unit 304 may also activate the remote notification function by, for example, transmitting a command signal indicating activation of the notification function to the remote operation support device 200 via the communication device 60. In this case, when the remote operation support device 200 (control device 210) receives a command signal from the work machine 100 via the communication device 220, it may output a visual or auditory alarm via the output device 230. This allows an operator or monitor who remotely operates or remotely monitors the work machine 100 via the remote operation support device 200 to know that a monitored object has entered the notification range around the work machine 100.

The remote notification function of the safety control unit 304 may be transferred to the remote operation support device 200 (for example, the notification control unit 2103 described below). In this case, the remote operation support device 200 receives information on the detection status of the monitored object by the object detection unit 303 from the work machine 100, determines whether the monitored object has entered the notification range based on the received information, and activates the external notification function if a monitored object is present within the notification range.

The safety control unit 304 may also vary the notification mode (i.e., the method of notification) depending on the positional relationship between the monitored object detected within the notification range and the work machine 100.

For example, when a monitored object detected within the notification range by the object detection unit 303 is located relatively far from a predetermined part of the work machine 100, such as the upper rotating body 3 or the hook HK (or the suspended load), the safety control unit 304 may output a relatively low-urgency alarm (hereinafter, "alarm of caution level") that calls attention to the monitored object. Hereinafter, the range of the notification range in which the distance to a predetermined part of the work machine 100, such as the upper rotating body 3 or the hook HK (or the suspended load), is relatively far, that is, the range corresponding to the alarm of the caution level, may be referred to as the "alarm notification range" for convenience. On the other hand, when a monitored object detected within the notification range by the object detection unit 303 is located relatively close to a predetermined part of the work machine 100, such as the upper rotating body 3 or the hook HK (or the suspended load), the safety control unit 304 may output a relatively high-urgency alarm (hereinafter, "alarm of caution level") that notifies the user that the monitored object is approaching a predetermined part of the work machine 100 and that the danger is increasing. Hereinafter, the range of the notification range that is relatively close to a specific part of the work machine 100, such as the upper rotating body 3 or the hook HK (or the suspended load), i.e., the range that corresponds to an alert level warning, may be referred to as the "alert notification range."

In this case, the safety control unit 304 may change the pitch, sound pressure, tone, and blowing period of the sound output from the output device (sound output device) between the warning level alarm and the alert level alarm. The safety control unit 304 may also change the color, shape, size, blinking status, and blinking period of the image indicating that a monitored object has been detected and the image (e.g., a frame or marker, etc.) that highlights the monitored object or its position on the surrounding image displayed on the display device between the warning level alarm and the alert level alarm. This allows the controller 30 to allow the operator, etc. to grasp the degree of urgency, in other words, the proximity of the monitored object to a specified part of the work machine 100, based on the difference in the alarm sound (alarm sound) output from the sound output device and the alarm image displayed on the display device.

After the alarm function starts to operate, if the monitored object that was detected by the object detection unit 303 is no longer detected within the monitored area, the safety control unit 304 may stop the alarm function. In addition, after the alarm function starts to operate, if a predetermined operation to deactivate the alarm function is received via the input device 52, the safety control unit 304 may stop the alarm function.

In addition, the safety control unit 304 activates the operation restriction function when, for example, a monitored object is detected by the object detection unit 303 within a predetermined range (hereinafter, "operation restriction range") included in the monitoring area. The operation restriction range is set to a range in which its outer edge is relatively closer to a predetermined part of the work machine 100, such as the upper revolving body 3 or the hook HK (or the suspended load), than the above-mentioned notification range. As a result, the safety control unit 304 can, for example, first activate the notification function when a monitored object enters the notification range from the outside, and then further activate the operation restriction function when the monitored object enters the inner operation restriction range. Therefore, the controller 30 can activate the notification function and the operation restriction function in stages in accordance with the movement of the monitored object to the inside within the monitoring area. Specifically, the safety control unit 304 may activate the notification function when a monitored object is detected within the notification range in which the distance D from a predetermined part of the work machine 100 is within a threshold Dth3 (<Dth2). In this case, the threshold value Dth3 may be constant regardless of the direction as viewed from a specific part of the work machine 100, or may change depending on the direction as viewed from a specific part of the work machine 100.

The operating limit range also includes at least one of an operating deceleration range in which the operating speed of the work machine 100 is slower than normal in response to an operating command corresponding to the operation of the operating device 26, remote operation, or automatic driving function, and an operating stop range in which the operation of the work machine 100 is stopped and maintained in a stopped state regardless of an operating command corresponding to the operation of the operating device 26, remote operation, or automatic driving function. For example, when the operating limit range includes both the operating deceleration range and the operating stop range, the operating stop range is a range within the operating limit range that is close to a specified part of the work machine 100, such as the upper rotating body 3 or the hook HK (or the suspended load). The operating deceleration range is a range within the operating limit range that is set outside the operating stop range.

The safety control unit 304 activates an operation limiting function that limits the operation of the work machine 100 by controlling the hydraulic control valve 31. In this case, the safety control unit 304 may limit the operation of all driven elements (i.e., the corresponding hydraulic actuators), or may limit the operation of some of the driven elements (hydraulic actuators). This allows the controller 30 to slow down or stop the operation of the work machine 100 when a monitored object is present around the work machine 100. Therefore, the controller 30 can suppress the occurrence of contact between the monitored object around the work machine 100 and the work machine 100 or the suspended load. In addition, the safety control unit 304 may activate the operation limiting function (operation stop function) by controlling an electromagnetic switching valve (not shown) of the pilot line 25 and blocking the pilot line 25.

Furthermore, the safety control unit 304 may stop the operation restriction function when the monitored object that was detected by the object detection unit 303 is no longer detected after the operation of the operation restriction function starts. Furthermore, the safety control unit 304 may stop the operation restriction function when a predetermined operation for deactivating the operation of the operation restriction function is received through the input device 52 after the operation of the operation restriction function starts. The operation for deactivating the alarm function on the input device 52 and the operation for deactivating the operation of the operation restriction function may be the same or different.

The safety control unit 304 may also have its function switched between ON (enabled) and OFF (disabled) in response to a predetermined operation by an operator or the like on the input device 52.

<Configuration of remote operation support device>
As shown in FIG. 5 , the remote operation assistance device 200 includes a control device 210 , a communication device 220 , an output device 230 , and an input device 240 .

The control device 210 performs various controls related to the remote operation support device 200. The functions of the control device 210 are realized by any hardware or any combination of hardware and software. The control device 210 is configured around a computer including, for example, a CPU, a memory device such as RAM, a non-volatile auxiliary storage device such as ROM, and various interface devices for input and output.

The control device 210 includes, as functional units, a remote operation support unit 2101, a display control unit 2102, a notification control unit 2103, a communication status monitoring unit 2104, and an operation status monitoring unit 2105. The functions of the remote operation support unit 2101, the display control unit 2102, the notification control unit 2103, the communication status monitoring unit 2104, and the operation status monitoring unit 2105 are realized, for example, by loading a program installed in an auxiliary storage device into a memory device and executing the program on a CPU.

The communication device 220 (an example of a communication device for an external device) is connected to the communication line NW and communicates with the outside of the remote operation support device 200 (e.g., the work machine 100).

The output device 230 (an example of an alarm device) outputs various information to the user of the remote operation support device 200. The output device 230 includes a display device 230A and a sound output device 230B.

Under the control of the control device 210, the display device 230A visually outputs various information related to the remote operation support device 200 to the user of the remote operation support device 200. Specifically, the display device 230A displays an information image related to the remote operation support device 200. The display device 230A includes, for example, a liquid crystal display or an organic EL display.

The output device 230 may include a lighting device that outputs various information to the user of the remote operation support device 200 in a visual manner. The lighting device may include, for example, a warning lamp.

The sound output device 230B includes a sound output device that outputs various information to the user of the remote operation support device 200 in an auditory manner. The sound output device includes, for example, a buzzer, a speaker, etc.

The input device 240 accepts input from a user of the remote operation support device 200 and outputs a signal representing the content of the input (e.g., operation input, voice input, gesture input, etc.). The signal representing the content of the input is taken into the control device 210. The user of the remote operation support device 200 includes an administrator of the remote operation support device 200, an operator of the work machine 100, a supervisor of the work machine 100, etc.

The input device 240 includes, for example, a remote control device 240A.

The remote control device 240A is used for remote control of the work machine 100 by an operator or for remote control of the work machine 100 by a supervisor intervening in the automatic driving function. The remote control device 240A accepts inputs from the operator or supervisor for remote control of multiple actuators (driven elements) and outputs a signal representing the content of the input to the control device 210.

The remote operation support unit 2101 performs control related to the remote operation of the work machine 100. Specifically, the remote operation support unit 2101 may take in an input signal related to the remote operation of the work machine 100 received by the remote operation device 240A, and transmit a remote operation signal indicating the content of the remote operation of the work machine 100 to the work machine 100 using the communication device 220.

The display control unit 2102 controls the display device 230A and causes the display device 230A to display various information images.

The display control unit 2102, for example, causes the display device 230A to display an image (peripheral image) showing the surroundings of the work machine 100 that is uploaded from the work machine 100.

Specifically, the display control unit 2102 causes the display device 230A to display a peripheral image corresponding to the imaging range of the imaging device 42. This allows the user (operator) of the remote operation support device 200 to remotely operate the work machine 100 while checking the movement of the attachment AT by visually checking the peripheral image corresponding to the imaging range of the imaging device 42. In addition, the user (monitor) of the remote operation support device 200 can monitor the work status of the work machine 100 while checking the movement of the attachment AT by visually checking the peripheral image corresponding to the imaging range of the imaging device 42.

The display control unit 2102 also causes the display device 230A to display a peripheral image corresponding to the imaging range of the imaging device 40. This allows the user of the remote operation support device 200 to visually check the peripheral image corresponding to the imaging range of the imaging device 40 and thereby check the situation around the work machine 100, which is difficult to check only from the peripheral image corresponding to the imaging range of the imaging device 42. Therefore, the remote operation support device 200 can improve the safety of the work machine 100 and the work site where the work machine 100 is used.

The display control unit 2102 may generate a processed image as a peripheral image based on the captured image of the imaging device 40 uploaded from the work machine 100, and display the processed image on the display device 230A.

When a command signal indicating the operation of the notification function (remote notification function) is received via the communication device 220, the notification control unit 2103 notifies the operator or supervisor via the output device 230 that a monitored object has been detected within the notification range around the work machine 100.

The notification control unit 2103, for example, controls the sound output device 230B to auditorily notify the operator or supervisor that a monitored object has been detected within the notification range around the work machine 100. At this time, the notification control unit 2103 may vary the pitch, sound pressure, tone, blowing period when blowing a sound periodically, sound content, etc. of the output sound according to various conditions.

The notification control unit 2103 may also visually notify the operator or the supervisor that a monitored object has been detected within the notification range around the work machine 100. Specifically, the notification control unit 2103 may control the display device 230A through the display control unit 2102 to display an image indicating that a monitored object has been detected on the display device 230A. The notification control unit 2103 may also highlight the monitored object shown in the peripheral image displayed on the display device 230A or the position on the peripheral image corresponding to the detected monitored object through the display control unit 2102. More specifically, the notification control unit 2103 may superimpose a frame surrounding the detected monitored object on the peripheral image displayed on the display device 230A through the display control unit 2102, or superimpose a marker on a position on the peripheral image corresponding to the actual position of the detected monitored object. This allows the display device 230A to realize a visual notification function for the operator or the supervisor. The notification control unit 2103 may also use a lighting device such as a warning light to notify an operator or a supervisor that a monitored object has been detected within the notification range.

The communication status monitoring unit 2104 monitors the communication status between the work machine 100 and the remote operation support device 200.

Specifically, the communication status monitoring unit 2104 monitors the communication status between the work machine 100 and the remote operation support device 200 based on the presence or absence of a predetermined signal received from the work machine 100 and the time required from transmission to reception of the predetermined signal. The communication status between the work machine 100 and the remote operation support device 200 includes, for example, the occurrence of communication interruption between the work machine 100 and the remote operation support device 200. In addition, the communication status between the work machine 100 and the remote operation support device 200 includes, for example, the occurrence of communication delay (the magnitude of communication delay) when a signal is transmitted from the work machine 100 to the remote operation support device 200. The predetermined signal is, for example, a response signal from the work machine 100 to a signal requesting a response that is periodically transmitted to the work machine 100 by the communication status monitoring unit 2104 through the communication device 220. In addition, the predetermined signal may be, for example, a signal corresponding to data (for example, a peripheral image) periodically transmitted (uploaded) from the work machine 100. This allows the communication status monitoring unit 2104 to monitor whether a communication interruption has occurred based on whether a specific signal has been received. The communication status monitoring unit 2104 can also measure the magnitude of a communication delay based on data (timestamp) indicating the transmission timing included in the specific signal and the time data when the specific signal was actually received through the communication device 220.

The operation status monitoring unit 2105 monitors the operation status of the actuator (driven element) of the work machine 100 based on the input status received by the remote control device 240A.

[Setting Processing of Notification Range]
Next, the process of setting the notification range by the control device 210 will be described with reference to FIGS.

<First Example of Notification Range Setting Process>
Fig. 8 is a flowchart showing in outline a first example of the processing for setting the notification range by the control device 210. This flowchart is repeatedly executed at a predetermined cycle while the work machine 100 is in operation, that is, from the start-up of the work machine 100 (e.g., key switch is turned on) to the stop of the work machine 100 (e.g., key switch is turned off). The same may be applied to the flowcharts in Figs. 9 and 10 described below.

As shown in FIG. 8, in step S102, the communication status monitoring unit 2104 monitors the communication status between the work machine 100 and the remote operation support device 200 based on the reception status of the above-mentioned specified signal and the time lag from transmission to reception of the specified signal.

When the control device 210 completes processing of step S102, it proceeds to step S104.

Based on the monitoring result, the communication status monitoring unit 2104 determines whether the communication delay between the work machine 100 and the remote operation support device 200 exceeds a predetermined standard. If the communication delay between the work machine 100 and the remote operation support device 200 does not exceed the predetermined standard, the communication status monitoring unit 2104 proceeds to step S106, and if the communication delay exceeds the predetermined standard, the communication status monitoring unit 2104 proceeds to step S108.

In step S106, the communication status monitoring unit 2104 sets the notification range of the notification function by the safety control unit 304 to the default state.

When the control device 210 completes processing of step S106, it proceeds to step S110.

On the other hand, in step S108, the communication status monitoring unit 2104 sets the notification range of the notification function by the safety control unit 304 to a state expanded outward from the default.

For example, if the communication delay between the work machine 100 and the remote operation support device 200 becomes large, the time lag from when a monitored object is captured in the notification range around the work machine 100 by the imaging device 40 or the like to when the notification function of the remote operation support device 200 is activated becomes large. Therefore, when the notification function of the remote operation support device 200 is activated and the operator or monitor confirms the presence of the monitored object, the monitored object may move inside the notification range and come even closer to the work machine 100 or the suspended load, etc. This may reduce the safety of the work machine 100 and the work site.

On the other hand, for example, in the case of an automobile (see, for example, JP 2016-71585, etc.), when the communication delay between the work machine 100 and the remote operation support device 200 becomes large, it is possible to automatically activate the operation restriction function to ensure safety.

However, while the main function of an automobile is to move at relatively high speeds, and deceleration or stopping is essential to ensure safety, the main function of a work machine is to work in a relatively small area, and there may be cases where deceleration or stopping is not required across the board. Therefore, if the operation restriction function is activated across the board, there is a possibility that the operation of the work machine 100 will be decelerated or stopped even in situations where safety has not been significantly reduced. As a result, there is a problem in terms of achieving both the safety of the work machine 100 and the operability of the work machine 100.

In addition, for example, when communication delays between the work machine 100 and the remote operation support device 200 become large, it is possible to activate the operation restriction function only when a monitored object is detected in the monitoring area around the work machine 100.

However, even if a monitored object is detected in the monitoring area around the work machine 100, there may be cases where safety is not significantly reduced, for example, when the monitored object is simply moving near the outer edge of the monitoring area. Therefore, simply activating the operation restriction function of the work machine 100 only when a monitored object is detected in the monitoring area around the work machine 100 may not be sufficient in terms of ensuring the operability of the work machine 100.

In response to these, in this example, the control device 210 can expand the notification range outward from the default. Therefore, even in a situation where there is a large communication delay between the work machine 100 and the remote operation support device 200, the control device 210 can activate the notification function in the remote operation support device 200 before the monitored object approaches within a relatively close range of the work machine, a suspended load, etc. Therefore, the control device 210 can improve the safety of the work machine 100 while suppressing a decrease in the workability (work efficiency) of the work machine 100.

For example, the communication status monitoring unit 2104 sets the outer edge of the notification range to a state in which it is expanded outward by a predetermined amount or ratio. In this case, the amount of expansion or the expansion ratio of the outer edge of the notification range may be uniform or may differ depending on the location, and there may be some locations that are expanded and some locations that are not expanded.

In addition, for example, the communication status monitoring unit 2104 may set the outer edge of the notification range to be expanded outward so that the amount of change (expansion amount) increases as the communication delay between the work machine 100 and the remote operation support device 200 increases.

When the control device 210 completes processing of step S108, it proceeds to step S110.

In step S110, the communication status monitoring unit 2104 transmits a signal to the work machine 100 via the communication device 220 requesting that the notification range be set to the range set in step S106 or step S108. As a result, the controller 30 (safety control unit 304) sets the notification range in response to the signal received from the remote operation support device 200 via the communication device 60. This allows the control device 210 to reflect the settings of steps S106 and S108 in the control of the actual notification function of the work machine 100.

When the control device 210 completes the processing of step S110, it ends the processing of this flowchart.

<Second Example of Notification Range Setting Process>
FIG. 9 is a flowchart illustrating a second example of the process of setting the notification range by the control device 210. In FIG.

As shown in FIG. 9, steps S202 and S204 are the same as steps S102 and S104 in FIG. 8, so their explanation is omitted.

In step S204, if the communication delay between the work machine 100 and the remote operation support device 200 does not exceed a predetermined standard, the communication status monitoring unit 2104 proceeds to step S206, and if the communication delay exceeds the predetermined standard, the communication status monitoring unit 2104 proceeds to step S208.

Step S206 is the same as step S106 in FIG. 8, so the explanation is omitted.

When the processing of step S206 is completed, the control device 210 proceeds to step S212.

On the other hand, in step S208, the operation status monitoring unit 2105 monitors the operation status of the work machine 100 based on the input status of the remote control device 240A.

When the processing of step S208 is completed, the control device 210 proceeds to step S210.

In step S210, the communication status monitoring unit 2104 sets the notification range of the notification function by the safety control unit 304 to a state expanded outward from the default in accordance with the operation state of the work machine 100 based on the monitoring result of step S202.

For example, when the undercarriage 1 is being operated to travel through the remote control device 240A, the communication status monitoring unit 2104 may set the outer edge of the notification range in the travel direction of the work machine 100 to a relatively larger, outwardly expanded state than the outer edge in other directions. This is because, as the work machine 100 moves, the distance between the work machine 100 and a monitored object that exists in the movement direction (travel direction) of the work machine 100 becomes relatively small.

For example, when the upper rotating body 3 is being rotated through the remote control device 240A, the communication status monitoring unit 2104 may set the outer edge of the notification range in the rotation direction around the rotation axis based on the orientation of the attachment AT to a state in which it is relatively expanded outward compared to the outer edge in the opposite direction to the rotation direction. This is because, as the work machine 100 rotates, the distance between the attachment AT and a monitored object that exists in the rotation direction around the rotation axis becomes relatively small.

When the control device 210 completes processing of step S210, it proceeds to step S212.

In step S212, the communication status monitoring unit 2104 transmits a signal to the work machine 100 via the communication device 220 requesting that the notification range be set to the range set in step S206 or step S210. As a result, the controller 30 (safety control unit 304) sets the notification range in response to the signal received from the remote operation support device 200 via the communication device 60. This allows the control device 210 to reflect the settings of steps S206 and S210 in the control of the actual notification function of the work machine 100.

<Third Example of Notification Range Setting Process>
FIG. 10 is a flowchart illustrating a third example of the process of setting the notification range by the control device 210. In FIG.

As shown in FIG. 10, steps S302 and S304 are the same as steps S102 and S104 in FIG. 8, so their explanation is omitted.

In step S304, if the communication delay between the work machine 100 and the remote operation support device 200 does not exceed a predetermined standard, the communication status monitoring unit 2104 proceeds to step S306, and if the communication delay exceeds the predetermined standard, the communication status monitoring unit 2104 proceeds to step S308.

Step S306 is the same as step S106 in FIG. 8, so its explanation is omitted.

When the control device 210 completes processing of step S306, it proceeds to step S314.

On the other hand, in step S308, the communication status monitoring unit 2104 determines whether or not a monitored object has been detected by the object detection unit 303 of the work machine 100. The communication status monitoring unit 2104 can determine whether or not a monitored object has been detected based on data indicating the object detection status successively received from the work machine 100 via the communication device 220. If a monitored object has been detected by the object detection unit 303, the communication status monitoring unit 2104 proceeds to step S310, and if a monitored object has not been detected, the communication status monitoring unit 2104 proceeds to step S312.

In step S310, the communication status monitoring unit 2104 sets the outer edge of the notification range to a state expanded outward from the default in accordance with at least one of the position and movement status of the detected monitored object.

For example, the communication status monitoring unit 2104 may shift the outer edge of the notification range in the direction in which the detected monitored object is located farther outward than the outer edge in other directions. This is because there is a possibility that the monitored object may move toward the inside of the notification range.

Also, for example, when a monitored object is moving toward the inside of the notification range, the communication status monitoring unit 2104 may change the notification range to a greater extent outside the notification range than when the monitored object is not moving toward the inside of the notification range. This is because it is necessary to notify the operator or monitor of the presence of a monitored object moving toward the inside of the notification range at an earlier timing.

In addition, for example, the communication status monitoring unit 2104 may change the outer edge of the notification range more outward when the moving speed of the monitored object is relatively fast than when the moving speed of the monitored object is relatively slow. This is because it is necessary to notify the operator or monitor at an earlier timing of a monitored object that is moving faster inside the notification range.

When the control device 210 completes processing of step S310, it proceeds to step S314.

On the other hand, in step S312, the communication status monitoring unit 2104 sets the outer edge of the notification range to a state in which it is expanded outward from the default so that the degree of change (amount of change or rate of change) is relatively smaller than when a monitored object is detected.

When the control device 210 completes processing of step S312, it proceeds to step S314.

In step S314, the communication status monitoring unit 2104 transmits a signal to the work machine 100 via the communication device 220 requesting that the notification range be set to the range set in step S306, step S310, or step S312. As a result, the controller 30 (safety control unit 304) sets the notification range in response to the signal received from the remote operation support device 200 via the communication device 60. This allows the control device 210 to reflect the settings of steps S306, S310, and S312 in the control of the actual notification function of the work machine 100.

[Example of monitoring screen]
Next, specific examples of a screen (hereinafter, "monitoring screen") including an image of the surroundings of the work machine 100, which is displayed on the display device 230A, will be described with reference to Figs. 11 to 14.

<Monitoring screen when communication delay is relatively small>
Fig. 11 and Fig. 12 are diagrams showing an example of a monitoring screen including a peripheral image of the work machine 100, displayed on the display device 230A. Specifically, Fig. 11 and Fig. 12 are diagrams showing a specific example of a monitoring screen when the communication status monitoring unit 2104 determines that the communication delay between the work machine 100 and the remote operation support device 200 is equal to or less than a predetermined standard. More specifically, Fig. 11 is a diagram showing an example of a monitoring screen (monitoring screen 1100) displayed on the display device 230A, including a peripheral image of the excavator 100A corresponding to the imaging range of the imaging device 40. Fig. 12 is a diagram showing an example of a monitoring screen (monitoring screen 1200) displayed on the display device 230A, including a peripheral image of the crawler crane 100B corresponding to the imaging range of the imaging device 40.

In this example, in addition to the monitoring screens 1100 and 1200 in Figs. 11 and 12, the captured image of the imaging device 42 and the processed image generated based on the captured image are displayed in another display area of the display device 230A, or on another display device 230A among the multiple display devices 230A. The same applies to the case where the monitoring screens 1300 and 1400 in Figs. 13 and 14 described below are displayed on the display device 230A.

For example, as shown in FIG. 11, under the control of the display control unit 2102, the display device 230A displays a monitoring screen 1100 including a shovel image CG and a viewpoint conversion image EP arranged around the shovel image CG as a peripheral image of the shovel 100A. The shovel image CG and the viewpoint conversion image EP are arranged according to the positional relationship between the shovel 100A (upper rotating body 3) and the imaging range of the imaging device 40 (cameras 40B, 40L, 40R). This allows the operator to properly grasp the positional relationship between the shovel 100A and objects around the shovel 100A shown in the viewpoint conversion image EP through the monitoring screen 1100.

In this example, the viewpoint conversion image EP is composed of a combination of an overhead image BVP of the surrounding area adjacent to the shovel 100A as viewed from directly above, and a horizontal image HVP of the surrounding area as viewed horizontally from the shovel 100A, which is arranged around the overhead image BVP. The viewpoint conversion image EP is obtained by projecting the captured images of each of the cameras 40B, 40L, and 40R onto a spatial model, and then reprojecting the projected images onto the spatial model onto another two-dimensional plane. The spatial model is the projection target of the captured images in virtual space, and is composed of one or more planes or curved surfaces, including planes or curved surfaces other than the plane on which the captured images are located.

In addition, in this example, a line LN1 at a certain distance from the shovel 100A is superimposed on the viewpoint conversion image EP of the monitoring screen 1100. The line LN1 may represent, for example, the outer edge of the notification range by the safety control unit 304. This allows the operator or supervisor to properly grasp the distance relationship between the shovel 100A and the surrounding objects shown in the viewpoint conversion image EP.

In addition, in this example, the viewpoint conversion image EP of the monitoring screen 1100 shows the worker W in a location corresponding to the rear of the shovel image CG. This allows the operator or supervisor to recognize that the worker W is located (present) behind the shovel 100A (upper rotating body 3).

In this example, the worker W is outside the line LN1, which corresponds to the outer edge of the notification range. Therefore, the worker W is not the target of the visual notification function, and no image emphasizing the presence of the worker W (e.g., the frame FRM1 described below) is displayed.

The display control unit 2102 may display a monitoring screen including at least one of the captured images from the cameras 40B, 40L, and 40R on the display device 230A. That is, the display control unit 2102 may display all of the captured images from the cameras 40B, 40L, and 40R or the captured images from two cameras side by side on the display device 230A, or may display the captured image from any one of the cameras on the display device 230A. Typically, the display control unit 2102 may display the captured images from the cameras 40B and 40R side by side on the display device 230A. In this case, the display control unit 2102 may superimpose lines representing the outer edges of the monitoring area, the notification range, and the like on the monitoring screen including the captured images from the cameras 40B, 40L, and 40R on the display device 230A, as in the case of the viewpoint conversion image EP.

Also, for example, as shown in FIG. 12, under the control of the display control unit 2102, the display device 230A displays a monitoring screen 1200 including an image captured by the imaging device 40 as an image of the surroundings of the crawler crane 100B.

The monitoring screen 1200 (image captured by the imaging device 40) shows the hook HK at the end of the main hoisting rope 7B and the load SL slinging on the hook HK.

In addition, in this example, a line LN2 at a certain distance from the suspended load SL is displayed superimposed on the monitoring screen 1200. The line LN2 represents, for example, the outer edge of a warning range by the safety control unit 304. This allows the operator or monitor to properly grasp the positional relationship between the suspended load SL and surrounding objects shown on the monitoring screen 1200 (image captured by the imaging device 40).

In addition, in this example, the monitoring screen 1200 shows two obstacles OB around the suspended load SL. This allows the operator or supervisor to recognize that there are two obstacles OB around the suspended load SL.

In this example, both obstacles OB are located outside the line LN1 that represents the outer edge of the notification range. Therefore, neither of the obstacles OB is the target of the visual notification function, and no image (e.g., the frame FRM2 described below) that emphasizes the presence of the obstacle OB is displayed.

The display control unit 2102 may also cause the display device 230A to display, as the peripheral image, a viewpoint conversion image generated based on the image captured by the imaging device 40 of the crawler crane 100B.

<Monitoring screen when communication delay is relatively large>
Fig. 13 and Fig. 14 are diagrams showing another example of a monitoring screen including a peripheral image of the work machine 100, displayed on the display device 230A. Specifically, Fig. 13 and Fig. 14 are diagrams showing a specific example of a monitoring screen when the communication status monitoring unit 2104 determines that the communication delay between the work machine 100 and the remote operation support device 200 exceeds a predetermined standard. More specifically, Fig. 13 is a diagram showing another example (monitoring screen 1300) of a monitoring screen including a peripheral image of the excavator 100A corresponding to the imaging range of the imaging device 40, displayed on the display device 230A. Fig. 14 is a diagram showing an example (monitoring screen 1400) of a monitoring screen including a peripheral image of the crawler crane 100B corresponding to the imaging range of the imaging device 40, displayed on the display device 230A.

For example, as shown in FIG. 13, the display device 230A displays a monitoring screen 1300 including a shovel image CG and a viewpoint conversion image EP as a peripheral image of the shovel 100A under the control of the display control unit 2102, as in the example described above (FIG. 11).

In addition, in this example, a line LN1 corresponding to the outer edge of the notification range is superimposed on the viewpoint conversion image EP of the monitoring screen 1300, as in the example described above.

In this example, the line LN1 has uniformly changed outward from the example described above, that is, in a direction away from the shovel image CG. This is because the communication delay between the shovel 100A and the remote operation support device 200 exceeds a predetermined standard. This allows the control device 210 to understand, via the display device 230A, that the outer edge of the notification range has changed outward from normal (when the communication delay is relatively small). Furthermore, the control device 210 can understand, via the fact that the line LN1 on the display device 230A (monitoring screen 1300) has moved outward from normal, that the communication delay between the shovel 100A and the remote operation support device 200 has become relatively large.

In addition, in this example, the viewpoint conversion image EP of the monitoring screen 1300 shows the worker W in a location corresponding to the rear of the shovel image CG, as in the example described above.

In addition, in this example, a frame FRM1 surrounding the worker W is displayed superimposed on the viewpoint conversion image EP of the monitoring screen 1300. This is because the outer edge of the notification range is shifted outward, so that the worker W is now inside the line LN1. This allows the operator or monitor to become aware of the presence of the worker W at an earlier timing in a situation where there is a relatively large time lag between when the worker W is captured by the imaging device 40 and when the notification function of the remote operation support device 200 is activated.

The manner in which the line LN1 corresponding to the outer edge of the notification range changes with respect to the above-mentioned example may be varied according to the operating state of the shovel 100A, as described above. For example, when the shovel 100A is operated to travel toward the rear of the upper rotating body 3, the degree of change of the line LN1 with respect to the above-mentioned example may be greater in the rear (lower) part (curved part) of the shovel image CG than in the left and right parts (straight parts) of the shovel image CG. Also, for example, when the upper rotating body 3 of the shovel 100A is operated to rotate to the right, the degree of change of the line LN1 with respect to the above-mentioned example may be greater in the right part than in the left part of the shovel image CG. Also, the manner in which the line LN1 corresponding to the outer edge of the notification range changes with respect to the above-mentioned example may be varied according to the position and movement status of the monitored object (worker W) around the shovel 100A, as described above. For example, the degree of change of the line LN1 with respect to the above-mentioned example may be greater in the rear (lower) part of the shovel image CG where the worker W is present than in the left or right part of the shovel image CG where the worker W is not present. Also, for example, the degree of change of the line LN1 with respect to the above-mentioned example may be greater when the worker W is moving toward the shovel 100A than when the worker W is not moving toward the shovel 100A. Also, for example, the degree of change of the line LN1 with respect to the above-mentioned example may be greater when the moving speed of the worker W is relatively high than when it is relatively low.

In addition, in this example, the monitoring screen 1300 displays an image MSG1 that displays a message indicating that a relatively large communication delay is occurring between the excavator 100A and the remote operation support device 200. This allows the operator or monitor to understand that a relatively large communication delay is occurring between the excavator 100A and the remote operation support device 200, and that the line LN1 has changed from normal due to the effect of the communication delay.

Also, for example, as shown in FIG. 14, the display device 230A displays a monitoring screen 1400 including an image captured by the imaging device 40 as an image of the surroundings of the crawler crane 100B under the control of the display control unit 2102, as in the above example (FIG. 12).

The monitoring screen 1400 (image captured by the imaging device 40) shows the hook HK at the end of the main hoisting rope 7B and the load SL slinging on the hook HK, as in the example described above.

In addition, in this example, a line LN2 representing the outer edge of the notification range is displayed superimposed on the monitoring screen 1200, as in the example described above.

In this example, the line LN2 has moved outward, i.e., away from the load SL (hook HK), compared to the above example. This is because the communication delay between the crawler crane 100B and the remote operation support device 200 exceeds a predetermined standard. This allows the control device 210 to understand, via the display device 230A, that the outer edge of the notification range has moved outward compared to normal (when the communication delay is relatively small). Furthermore, the control device 210 can understand, via the fact that the line LN2 has moved outward compared to normal on the display device 230A (monitoring screen 1400), that the communication delay between the excavator 100A and the remote operation support device 200 has become relatively large.

In addition, in this example, the monitoring screen 1400 shows two obstacles OB around the suspended load SL, as in the example described above.

In addition, in this example, a frame FRM2 surrounding each of the two obstacles OB is displayed superimposed on the monitoring screen 1400. This is because the outer edge of the notification range changes outward, so that the obstacle OB is now present inside the line LN2. This allows the operator or monitor to become aware of the presence of the obstacle OB at an earlier timing in a situation where there is a relatively large time lag between when the obstacle OB is captured by the imaging device 40 and when the notification function is activated by the remote operation support device 200.

The manner in which the line LN2 corresponding to the outer edge of the notification range changes with respect to the above-mentioned example may be varied according to the operating state of the crawler crane 100B, as described above. For example, when the crawler crane 100B is operated to travel toward the front of the upper rotating body 3, the degree of change of the line LN2 with respect to the above-mentioned example may be greater in the front portion than in the left, right, and rear portions of the hook HK (suspended load SL). Also, for example, when the upper rotating body 3 of the crawler crane 100B is operated to rotate to the right, the degree of change of the line LN2 with respect to the above-mentioned example may be greater in the right portion than in the left portion of the hook HK (suspended load SL). Also, the manner in which the line LN2 corresponding to the outer edge of the notification range changes with respect to the above-mentioned example may be varied according to the position and movement status of the monitored object (obstacle OB) around the crawler crane 100B, as described above. For example, the degree of change of the line LN2 with respect to the above-mentioned example may be greater in the front or rear part of the hook HK (suspended load SL) where the obstacle OB is present than in the left or right part of the hook HK (suspended load SL) where the obstacle OB is not present. Also, for example, the degree of change of the line LN2 with respect to the above-mentioned example may be greater when the obstacle OB is moving relatively toward the hook HK (suspended load SL) than when the obstacle OB is not moving relatively toward the hook HK (suspended load SL). Also, for example, the degree of change of the line LN2 with respect to the above-mentioned example may be greater when the relative moving speed of the obstacle OB with respect to the hook HK (suspended load SL) is relatively high than when it is relatively low.

In addition, in this example, the monitoring screen 1400 displays an image MSG2 that displays a message indicating that a relatively large communication delay is occurring between the crawler crane 100B and the remote operation support device 200. This allows the operator or monitor to understand that a relatively large communication delay is occurring between the crawler crane 100B and the remote operation support device 200, and that the line LN2 has changed from normal due to the effect of that communication delay.

[Action]
Next, the operation of the remote operation support system SYS according to this embodiment will be described.

In this embodiment, the imaging device 40 of the work machine 100 acquires data (image data) regarding objects in the vicinity of the work machine 100. The communication device 60 of the work machine 100 communicates with the remote operation support device 200. The communication device 220 of the remote operation support device 200 communicates with the work machine 100. The remote operation device 240A of the remote operation support device 200 accepts input for remotely operating the work machine 100. The controller 30 of the work machine 100 or the control device 210 of the remote operation support device 200 detects a monitored object in the vicinity of the work machine 100 based on the output of the imaging device 40. The output device 230 is provided in the remote operation support device 200, and when a monitored object is detected within the notification range in the vicinity of the work machine 100 by the controller 30 or the control device 210 under the control of the control device 210, the output device 230 notifies the user using the remote operation device 240A of the detection. When the communication delay between the work machine 100 and the remote operation support device 200 is relatively large, the remote operation support system SYS shifts the outer edge of the notification range outward more than when the communication delay is relatively small.

As a result, the remote operation support system SYS can activate the notification function for the monitored object at a somewhat earlier stage than when the monitored object enters the range inside the notification range where safety may be reduced. Therefore, even in a situation where the time lag between when the monitored object is captured by the imaging device 40 and when the notification function is activated by the remote operation support device 200 is relatively large due to communication delay, the remote operation support system SYS can notify the monitor or operator of the presence of the monitored object before the monitored object enters the range inside the notification range where safety may be relatively reduced. This improves the safety of the work machine 100. In addition, the operator or monitor can check the monitoring screen displayed on the display device 230A in response to the activation of the notification function and determine for himself whether to continue or stop the operation of the work machine 100. Therefore, when the communication delay between the work machine 100 and the remote operation support device 200 becomes relatively large, it is possible to suppress a situation in which the workability of the work machine 100 is significantly reduced, such as when the operation restriction function is uniformly activated or the operation restriction range is expanded outward. Therefore, the remote operation support system SYS can improve the safety of the work machine 100 while suppressing a decrease in the workability (work efficiency) of the work machine 100.

In addition, in this embodiment, when the above-mentioned communication delay is relatively large, the remote operation support system SYS may change the outer edge of the notification range outward in accordance with the user's input to the remote operation device 240A.

As a result, in a situation where the above-mentioned communication delay is relatively large, the remote operation support system SYS can shift the outer edge of the notification range that has an impact on safety outward from the center in accordance with the operation state of the work machine 100 via the remote operation device 240A. Therefore, the remote operation support system SYS can more appropriately achieve a balance between the safety of the work machine 100 and the workability of the work machine 100.

In addition, in this embodiment, when the above-mentioned communication delay is relatively large, the remote operation support system SYS may change the outer edge of the reporting range in the traveling direction as seen from the work machine 100 to be significantly more outward than the outer edges of the reporting range in other directions when the work machine 100 is being operated to travel through the remote operation device 240A.

As a result, the remote operation support system SYS can specifically change the outer edge of the notification range that has an impact on the safety of the work machine 100 outward from the center in accordance with the operation state of the work machine 100 via the remote operation device 240A.

In addition, in this embodiment, when the above-mentioned communication delay is relatively large, the remote operation support system SYS changes the outer edge of the notification range in the rotation direction based on the orientation of the work attachment around the rotation axis to a greater extent outside than the outer edge of the notification range in the opposite direction to the rotation direction when the work machine 100 is being rotated through the remote operation device 240A.

As a result, the remote operation support system SYS can specifically change the outer edge of the notification range that has an impact on the safety of the work machine 100 outward from the center in accordance with the operation state of the work machine 100 via the remote operation device 240A.

In addition, in this embodiment, when the above-mentioned communication delay is relatively large, the remote operation support system SYS changes the outer edge of the reporting range depending on at least one of the position and movement status of the monitored object present in the vicinity of the work machine 100.

This allows the remote operation support system SYS to shift the outer edge of the reporting range that has an impact on safety outward from the center in accordance with the position and movement status of the monitored object. Therefore, the remote operation support system SYS can more appropriately balance the safety of the work machine 100 and the workability of the work machine 100.

In addition, in this embodiment, when the communication delay is relatively large, the remote operation support system SYS may change the outer edge of the notification range in the direction in which the monitored object is located as viewed from the work machine 100 to be more outward than the outer edges of the notification range in other directions.

This allows the remote operation support system SYS to specifically shift the outer edge of the reporting range that has an impact on safety outward from the center in accordance with the position of the monitored object.

In addition, in this embodiment, when the communication delay is relatively large, the remote operation support system SYS may change the outer edge of the notification range more outward when the monitored object is moving toward the inside of the notification range than when the monitored object is not moving toward the inside of the notification range.

This allows the remote operation support system SYS to specifically shift the outer edge of the reporting range that has an impact on safety outward, based on the center, in accordance with the movement status of the monitored object.

In addition, in this embodiment, when the communication delay is relatively large, the remote operation support system SYS may change the outer edge of the specified range more outward when the movement speed of the monitored object is relatively high than when the movement speed of the monitored object is relatively low.

This allows the remote operation support system SYS to specifically shift the outer edge of the reporting range that has an impact on safety outward, in accordance with the movement status of the monitored object.

In addition, in this embodiment, the display device 230A may display information about the notification range such that the display content changes in response to changes in the outer edge of the notification range.

As a result, the remote operation support system SYS can allow the operator or monitor to understand, via the display device 230A, the changes in the outer edge of the notification range that accompany the increase in communication delay described above.

Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the invention as described in the claims.

30 Controller (object detection device)
40 Imaging device (acquisition device)
40B camera 40L camera 40R camera 42 imaging device 42A camera 42B camera 50 output device 52 input device 60 communication device (communication device for work device)
100 Working machine 100A Shovel 100B Crawler crane 200 Remote operation support device (external device)
210 Control device (object detection device)
220 Communication device (communication device for external device)
230 Output device (alarm device)
230A Display device 230B Sound output device 240 Input device 240A Remote operation device 2101 Remote operation support unit 2102 Display control unit 2103 Notification control unit 2104 Communication status monitoring unit 2105 Operation status monitoring unit SYS Remote operation support system

Claims (10)

An acquisition device provided on a work machine for acquiring data relating to objects in the vicinity of the work machine;
a work machine communication device that is provided in the work machine and communicates with an external device that assists in remote operation of the work machine;
an external device communication device provided in the external device and configured to communicate with the work machine;
a remote operation device provided in the external device and configured to receive an input for remotely operating the work machine;
an object detection device that is provided on the work machine or the external device and detects a predetermined object around the work machine based on an output of the acquisition device;
an alarm device that is provided in the external device and that, when the predetermined object is detected by the object detection device within a predetermined range around the work machine, notifies a user of the remote control device of that fact,
When a communication delay between the work machine and the external device is relatively large, the outer edge of the predetermined range is shifted outward compared to when the communication delay is relatively small.
Remote operation support system. When the communication delay is relatively large, the outer edge of the predetermined range is shifted outward in accordance with the input contents of the user to the remote control device.
The remote operation support system according to claim 1 . When the communication delay is relatively large, when the work machine is being operated to travel through the remote control device, the outer edge of the specified range in the traveling direction as seen from the work machine is significantly shifted outward compared to the outer edges of the specified range in other directions.
The remote operation support system according to claim 2 . When the communication delay is relatively large, when a swing operation of the work machine is being performed through the remote control device, an outer edge of the predetermined range in a swing direction based on a direction of the work attachment about a swing axis is largely shifted outward from an outer edge of the predetermined range in a direction opposite to the swing direction.
The remote operation support system according to claim 2 or 3. when the communication delay is relatively large, an outer edge of the predetermined range is changed in accordance with at least one of a position and a movement state of the predetermined object present in the periphery of the work machine.
The remote operation support system according to any one of claims 1 to 4. when the communication delay is relatively large, the outer edge of the predetermined range in the direction in which the predetermined object is located as viewed from the work machine is shifted more outward than the outer edges of the predetermined range in other directions;
The remote operation support system according to claim 5 . When the communication delay is relatively large, when the predetermined object is moving toward the inside of the predetermined range, the outer edge of the predetermined range is changed more outwardly than when the predetermined object is not moving toward the inside of the predetermined range.
7. The remote operation support system according to claim 5 or 6. When the communication delay is relatively large, and the moving speed of the predetermined object is relatively large, the outer edge of the predetermined range is changed outwardly to a greater extent than when the moving speed of the predetermined object is relatively small.
The remote operation support system according to any one of claims 5 to 7. a display device provided in the external device and configured to display information about the predetermined range such that the display content changes in response to a change in the outer edge of the predetermined range;
The remote operation support system according to any one of claims 1 to 8. a communication device that communicates with a work machine having an acquisition device that acquires data related to objects around the work machine;
a remote control device that receives an input for remotely controlling the work machine;
a notification device that, when a predetermined object is detected within a predetermined range around the work machine based on the output of the acquisition device, notifies a user of the remote control device of that fact,
when a communication delay between the work machine and the work machine is relatively large, the outer edge of the predetermined range is changed in a direction away from the work machine compared to when the communication delay is relatively small.
Remote operation support device.

Images