JP7528892B2 - Work support system and work support complex system - Google Patents

Description

The present invention relates to a remote control system having a function for mutual communication between a work machine and a remote control device for remotely controlling the work machine.

For example, a work machine such as a hydraulic excavator has a lower traveling body and an upper rotating body that can rotate relative to the lower traveling body. In addition, the upper rotating body is provided with a cab that forms the operator's compartment.

In such a work machine, the upper rotating body rotates relative to the lower traveling body, so the front direction of the cab and the forward or reverse direction of the work machine may be different directions. This creates a problem in that it is difficult for the operator to know in which direction the work machine will move when operating the travelling mechanism.

Therefore, the direction of travel of the work machine is indicated by an arrow on a monitor installed inside the cab. For example, Patent Document 1 discloses a monitoring device for a construction machine in which an upper rotating body can rotate relative to a lower traveling body, and which is equipped with a camera that photographs the surroundings of the machine and displays images taken by the camera on a monitor screen installed inside the cab.

Patent No. 5473870

However, in the monitor device of Patent Document 1, the relative angle of the lower traveling body to the upper rotating body is estimated based on the time change of the image displayed on the monitor screen. This requires complex image processing to estimate the relative angle, which creates a problem of high processing load.

The present invention was made in consideration of these problems, and aims to provide a remote control system that reduces the processing load and can indicate the forward or reverse direction of a work machine with a simple configuration.

In order to achieve this object, the remote control system of the present invention is a remote control system having a work machine having a lower traveling body with a pair of left and right crawlers and an upper rotating body that can rotate relative to the lower traveling body, a remote control device for remotely operating the work machine, and a function of mutual communication with each other, characterized in that it has a first support processing element that recognizes the forward or backward direction of the lower traveling body based on the respective reflection patterns of a first marker that is different in shape from each other and is provided in front of the pair of crawlers and a second marker that is provided behind the pair of crawlers, which are unevenly arranged in the front-to-rear direction of each of the pair of crawlers in a first captured image acquired through a first imaging device that is provided on the upper rotating body and is provided so as to include at least one of the crawlers of the lower traveling body in its imaging area, and a second support processing element that transmits to the remote control device a composite image in which an indicator image representing the forward or backward direction of the lower traveling body recognized by the first support processing element is at least partially superimposed on the work machine or its surroundings in the first captured image.

According to the remote control system of the present invention, the first support processing element is able to recognize the forward or reverse direction of the pair of crawlers based on the manner in which the first marker and the second marker are reflected in the first captured image. This allows the second support processing element to generate a composite image in which the first captured image and an indicator image indicating the forward or reverse direction of the undercarriage are superimposed, and the composite image can be transmitted to the remote control device. The operator can easily grasp the forward or reverse direction of the undercarriage of the work machine by visually checking the composite image displayed on the remote control device.

In the remote control system of the present invention, the first support processing element recognizes the rotation angle of the upper rotating body relative to the lower running body based on the reflection state of the pair of crawlers that changes in response to the rotation of the upper rotating body relative to the lower running body and the reflection state of each of the first marker and the second marker, and it is preferable that the second support processing element transmits to the remote control device a composite image in which the indicator image is at least partially superimposed on the work machine or its surroundings in the first captured image at an angle corresponding to the rotation angle of the upper rotating body relative to the lower running body recognized by the first support processing element.

According to this aspect, the first support processing element can recognize the rotation angle of the upper rotating body relative to the lower traveling body, for example, based on the appearance of the crawler in the first captured image, i.e., the distinction between the left and right crawlers (one or both) captured in the first captured image, the inclination angle of the left and right crawlers relative to the imaging direction of the first imaging device, and the distinction between the first marker and the second marker captured in the first captured image. Therefore, for example, the second support processing element can generate a composite image by superimposing an indicator image corresponding to the inclination angle of the left and right crawlers relative to the imaging direction of the first imaging device, and can transmit the composite image to the remote control device. Therefore, the operator can easily recognize the forward or backward direction of the lower traveling body just by visually checking the composite image.

In the remote control system of the present invention, it is preferable that the second support processing element transmits to the remote control device a composite image in which an index image showing the forward or backward direction of the lower traveling body according to the rotation angle of the upper rotating body relative to the lower traveling body is at least partially superimposed on the work machine or its surroundings in the second captured image acquired through a second imaging device provided on the upper rotating body separately from the first imaging device.

According to this aspect, by transmitting a composite image in which an index image is superimposed on the second captured image to a remote control device, for example, an operator can perform operations without moving his or her line of sight while working. This can therefore improve the operator's work efficiency.

It is preferable that the first support processing element recognizes the rotation angle of the upper rotating body relative to the lower running body based on the distinction between the first marker and the second marker reflected in the first captured image and the number of the first marker and the second marker reflected in the first captured image.

According to this aspect, for example, when two first markers or two second markers are reflected in the first captured image, the rotation angle of the upper rotating body relative to the lower traveling body is recognized to be a specific angle, for example, an angle close to 0° or 180°. Also, for example, when one first marker (or one second marker) is reflected in the first captured image, the rotation angle of the upper rotating body relative to the lower traveling body is recognized to be a specific angle, for example, an angle close to 45° or 135° (225° or 315°). Furthermore, depending on the type of marker (first marker or second marker) reflected in the first captured image, it is possible to recognize the forward and backward directions of the pair of crawlers. Furthermore, when the first marker or the second marker is not reflected in the first captured image, the rotation angle of the upper rotating body relative to the lower traveling body is recognized to be a specific angle, for example, an angle close to 90°. Therefore, the first support processing element can recognize the rotation angle of the upper rotating body relative to the lower running body and the front-rear direction of the pair of crawlers without using the reflection manner of the pair of crawlers in the first captured image. This makes it unnecessary for the first support processing element to perform image analysis processing required for the reflection manner of the pair of crawlers, thereby reducing the processing load of the second support processing element.

In the remote control system of the present invention, it is preferable that the first support processing element recognizes the rotation angle of the upper rotating body relative to the lower traveling body based on the left-right position of the first marker or the second marker in the first captured image as the reflection mode of the first marker and the second marker contained in the first captured image.

The appearance of the first marker and the second marker included in the first captured image changes depending on the rotation angle of the upper rotating body relative to the lower running body. Specifically, the left-right position of the first marker or the second marker in the first captured image changes depending on the change in the rotation angle. Therefore, the first support processing element can recognize the rotation angle of the upper rotating body relative to the lower running body by recognizing the left-right positions of the first marker and the second marker in the first captured image.

In the remote control system of the present invention, it is preferable that the first support processing element acquires the first captured image including the first marker formed in a three-dimensional shape protruding from the lower traveling body or the second marker formed in a three-dimensional shape protruding from the lower traveling body.

According to this aspect, even if the first marker and the second marker are covered with mud or dirt, the first support processing element can easily recognize the first marker and the second marker.

In the remote control system of the present invention, it is preferable that the first imaging devices are arranged in multiple directions facing different directions in the rotation direction of the upper rotating body relative to the lower running body, the first support processing element recognizes the forward or backward direction of the lower running body based on the respective reflection patterns of the first marker and the second marker in the multiple first captured images acquired through the multiple first imaging devices, and the second support processing element transmits to the remote control device the composite image in which the indicator image is superimposed on the first captured image used by the first support processing element to recognize the forward or backward direction of the lower running body.

According to this aspect, even if the upper rotating body has a rotation angle relative to the lower running body such that the first marker or the second marker is not captured in the first captured image of one first imaging device, the first support processing element can recognize the forward or backward direction of the pair of crawlers by using another first captured image captured by another first imaging device.

In the remote control system of the present invention, it is preferable that the first support processing element acquires operation acceptance information, which is information indicating whether or not the remote control device is capable of accepting operations for the work machine, and the second support processing element transmits the composite image to the remote control device when the operation acceptance information indicates that the remote control device is capable of accepting operations for the work machine.

According to this aspect, the forward or reverse direction of the undercarriage can be displayed only when the operator operates the work machine, reducing the inconvenience felt by the operator.

In the remote control system of the present invention, it is preferable that the first support processing element acquires travel operation information of a preparatory operation, which is an operation representing an intention to input an operation to move the work machine forward or backward in the remote control device, and the second support processing element transmits to the remote control device the composite image generated using the indicator image representing the forward direction of the lower traveling body when the intention represented in the preparatory operation is the operation input to move forward, and transmits to the remote control device the composite image generated using the indicator image representing the backward direction of the lower traveling body when the intention represented in the preparatory operation is the operation input to move backward.

According to this aspect, the composite image with the superimposed index image can be displayed only when the operator operates the work machine to move forward or backward, reducing the annoyance of always having the composite image with the superimposed index image displayed.

The remote control system of the present invention is a remote control system having a communication function between a work machine having a lower traveling body with a pair of left and right crawlers, an upper rotating body that can rotate relative to the lower traveling body, and a remote control device for remotely operating the work machine, and the remote control system has a communication function between the work machine and a lower traveling body. The remote control system has a communication function between a work machine having a lower traveling body with a pair of left and right crawlers, an upper rotating body that can rotate relative to the lower traveling body, and a remote control device for remotely operating the work machine. In a first captured image acquired through a first imaging device that is provided on the upper rotating body and is provided so as to include at least one of the crawlers of the lower traveling body in an imaging area, the remote control system has a communication function between the work machine and a remote control device for remotely operating the work machine. The system is characterized by having a first support processing element that recognizes the forward or backward direction of the lower traveling body based on the respective reflection patterns of a first marker provided in front and a second marker provided behind the pair of crawlers, and a second support processing element that transmits to the remote control device a composite image in which an indicator image representing the forward or backward direction of the lower traveling body recognized by the first support processing element is at least partially superimposed on the work machine or its surroundings on a second captured image acquired through a second imaging device provided on the upper rotating body separately from the first imaging device.

1 is a diagram illustrating a configuration of a work support system according to an embodiment of the present invention; FIG. 2 is an explanatory diagram of the configuration of the remote control device of FIG. 1 . FIG. 2 is an explanatory diagram relating to the configuration of the work machine of FIG. 1. 13 is a flowchart showing an example of a work support process performed by the work support server. 5 is a flowchart showing an example of a forward/reverse direction calculation process shown in FIG. 4 . FIG. 2 is an explanatory diagram showing an example of an image captured by an imaging device. FIG. 2 is an explanatory diagram showing an example of an image captured by an imaging device. FIG. 2 is an explanatory diagram showing an example of an image captured by an imaging device. FIG. 2 is an explanatory diagram showing an example of an image captured by an imaging device. FIG. 2 is an explanatory diagram showing an example of an image captured by an imaging device. 13 is a flowchart showing another example of the work support process by the work support server.

(Configuration of remote control system)
The work support system as one embodiment of the present invention shown in Fig. 1 is configured with a work support server 10 for supporting the remote operation of a plurality of work machines 40 by a remote operation device 20. The work support server 10, the remote operation device 20, and each of the plurality of work machines 40 are configured to be able to communicate with each other via a common or separate network. Note that in this figure, the work support server 10 is connectable to a plurality of work machines 40, but it is sufficient that the work support server 10 is connectable to one or more work machines 40, and there is no limit to the number of connected machines.

In this embodiment, the work support server 10 is described as a remote operation system having a function of mutual communication between each of a plurality of work machines 40 and a remote operation device 20 for remotely operating one of the plurality of work machines 40 that is to be remotely operated.

The components (hardware) of the present invention "obtain" various pieces of information, which encompasses any and all arithmetic processing to prepare various pieces of information in a form that can be used in subsequent arithmetic processing, such as receiving the information, reading or searching the information from an internal storage device (e.g., memory) and/or an external storage device (e.g., an external database server), and performing arithmetic processing on the received, read, searched, etc. information to calculate, estimate, predict, identify, etc. the information.

(Configuration of the work support server)
The work support server 10 includes a remote operation support device 100 including a first support processing element 101 and a second support processing element 102 , a database 110 , and a server wireless communication device 122 .

The first support processing element 101 and the second support processing element 102 are composed of a calculation processing device (a single-core processor or a multi-core processor or a processor core that constitutes the same), read the necessary data and software from a storage device such as a memory, and execute the calculation processing described below according to the software on the data.

The database 110 stores captured image data, etc. The database 110 may be configured as a database server separate from the work support server 10.

(Configuration of remote control device)
The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.

The remote input interface 210 includes a remote control mechanism 211 and a third imaging device 212.

The remote control mechanism 211 includes a traveling operation device, a slewing operation device, a boom operation device, an arm operation device, a bucket operation device, and a hydraulic lock lever (shutoff lever) that switches the state of acceptance of operations by the operator. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the traveling operation device is operated to move the lower traveling body 430 having a pair of left and right crawlers of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (slewing lever) of the slewing operation device is operated to move the hydraulic slewing motor that constitutes the slewing mechanism 440 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 462 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder 464 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder 466 of the work machine 40. The cutoff lever is operated to switch the state of the pilot pressure valve of the work machine 40.

The operating levers constituting the remote control mechanism 211 are arranged around the seat St on which the operator OP1 sits, as shown in FIG. 2, for example. The seat St is in the form of a high-back chair with armrests, but it may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or any other form of seating on which the operator OP1 can sit.

A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operating lever may serve as multiple operating levers. For example, the left operating lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 may function as an arm lever when operated in the forward/backward direction, and as a rotation lever when operated in the left/right direction. Similarly, the right operating lever 2112 provided in front of the right frame of the seat St shown in FIG. 2 may function as a boom lever when operated in the forward/backward direction, and as a bucket lever when operated in the left/right direction. The lever pattern may be changed as desired by the operation command of the operator OP1.

The cut-off lever 2113, which is provided below the left operating lever 2111 in front of the left frame of the seat St, functions as an operating lever to release the lock when it is raised (turned OFF) so that the work machine 40 will not move even if the operating levers 2110, 2111, 2112 are operated, and to release the lock when it is lowered (turned ON).

The remote output interface 220 includes a remote image output device 221 and a remote wireless communication device 222.

As shown in FIG. 2, the remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, which are respectively arranged in front of the seat St, diagonally forward to the left, and diagonally forward to the right. The shapes and sizes of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different.

The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or inclined to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of multiple image output devices divided into multiple parts. For example, the central remote image output device 2210 may be composed of a pair of image output devices adjacent to each other above and below, each having a roughly rectangular screen.

The remote audio output device 222 is composed of one or more speakers, and is composed of a central remote audio output device 2220, a left remote audio output device 2221, and a right remote audio output device 2222, which are arranged behind the seat St, at the rear of the left armrest, and at the rear of the right armrest, respectively, as shown in FIG. 2. The specifications of the central remote audio output device 2220, the left remote audio output device 2221, and the right remote audio output device 2222 may be the same or different.

The third imaging device 212 is provided so as to be able to capture images of, for example, the seat St and the remote control mechanism 211. In this embodiment, as also shown in FIG. 2, the third imaging device 212 is provided at the upper end side of the central remote image output device 2210 with its optical axis directed toward the seat St. In other words, the third imaging device 212 is installed so as to be able to capture images of the actions of the operator OP1 to operate the remote control mechanism 211 when the operator OP1 is seated in the seat St.

(Configuration of the work machine)
1, the work machine 40 is equipped with an actual machine control device 400, an actual machine input interface 410, and an actual machine output interface 420. The actual machine control device 400 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.

The work machine 40 is, for example, a hydraulic, electric, or hybrid drive crawler excavator (construction machine) that combines hydraulic and electric drive systems, and as shown in FIG. 3, includes a lower running body 430 equipped with a pair of left and right crawlers, and an upper rotating body 450 that is mounted on the lower running body 430 so as to be rotatable via a rotating mechanism 440. A cab 454 (operator's compartment) is provided on the front left side of the upper rotating body 450. An operating mechanism 460 is provided in the front center of the upper rotating body 450.

As shown in FIG. 6B, a first marker M1 is provided in front of each of the pair of crawlers 430a, 430b of the work machine 40. Also, as shown in FIG. 10C, a second marker M2 is provided behind each of the pair of crawlers 430a, 430b of the work machine 40.

The first marker M1 and the second marker M2 may be markers that can be distinguished from each other, such as two-dimensional codes that hold front-back information, figures such as circles and triangles, or symbols. Furthermore, the first marker M1 and the second marker M2 may be markers that can be distinguished from each other, such as steps that protrude outward from the crawler (steps for the operator to place his/her feet when entering the cab 454), convex parts, or three-dimensional shapes such as concave parts that are recessed from the crawler. For example, if the first marker M1 and the second marker M2 are covered with mud or the like, it is difficult to perform image recognition of symbols, figures, or the like, but if the markers are three-dimensional, image recognition is possible as long as a form corresponding to the three-dimensional shape is displayed.

That is, the first marker M1 and the second marker M2 are unevenly arranged in the front-to-rear direction of each of the pair of crawlers 430a, 430b, and have different shapes. The first marker M1 and the second marker M2 are stored in the database 110 as image data.

The real machine input interface 41 includes a real machine operation mechanism 411, first imaging devices 412a and 412b, and a second imaging device 412c. The real machine operation mechanism 411 includes multiple operation levers arranged around a seat inside the cab 454 in the same manner as the remote control mechanism 211.

A drive mechanism or robot is provided in the cab 454 that receives a signal corresponding to the operation mode of the remote control lever and moves the actual operation lever based on the received signal.

The first imaging device (hereinafter also referred to as the foot camera) 412a is provided on the upper rotating body 450 as shown in FIG. 3, and is provided so as to include at least one of the crawlers of the lower traveling body 430 in the imaging area. In this embodiment, the foot camera 412a is provided on the bottom surface (outer wall surface) of the cab 454 of the upper rotating body 450. Specifically, the foot camera 412a is provided on the bottom surface (outer wall surface) of the cab 454 with the light receiving element (imaging direction) facing the front direction where the front window of the cab 454 is provided. The foot camera 412a may be provided, for example, on the base end side of the boom foot.

It is preferable that a plurality of first imaging devices are arranged facing different directions in the rotation direction of the upper rotating body 450 relative to the lower running body 430. In this embodiment, a first imaging device (hereinafter also referred to as a left camera) 412b is provided in addition to the foot camera 412a. The foot camera 412a and the left camera 412b are installed with their optical axes facing in different directions. It is even more preferable that the imaging area of the left camera 412b is an area that is not included in the imaging area of the foot camera 412a.

In other words, the foot camera 412a and the left camera 412b are provided so that at least one of the foot camera 412a and the left camera 412b includes at least one of the first marker M1 and the second marker M2 in the imaging area regardless of the angle of the upper rotating body 450 relative to the lower running body 430. For example, it is preferable that the angle between the optical axis directions of the foot camera 412a and the left camera 412b is approximately 90°. In this embodiment, the left camera 412b is provided on the outer wall surface of the cab 454 with the light receiving element (imaging direction) facing in the direction in which the side window on the left side of the cab 454 is provided.

The left camera 412b may be provided on the outer wall surface of the cab 454 with the light receiving element (imaging direction) facing in the direction where the side window on the right side of the cab 454 is provided. The left camera 412b may also be provided on the roof of the cab 454, or may be provided so as to be able to capture images of the outside (e.g., the right side or left side) from inside the cab 454.

The second imaging device 412c (hereinafter also referred to as the main camera) is provided on the upper rotating body 450 separately from the foot camera 412a and the left camera 412b. The main camera 412c is installed, for example, inside the cab 454, and captures an image of the environment including at least a part of the operating mechanism 460 through the front window and a pair of left and right side windows. The captured image may omit some or all of the front window (or window frame) and the side windows.

The actual machine status sensor group 414 is composed of angle sensors for measuring the rotation angle (raising and lowering angle) of the boom 461 relative to the upper rotating body 450, the rotation angle of the arm 463 relative to the boom 461, and the rotation angle of the bucket 465 relative to the arm 463, a rotation angle sensor for measuring the rotation angle of the upper rotating body 450 relative to the lower running body 430, an external force sensor for measuring the external force acting on the bucket 465, and a three-axis acceleration sensor for measuring the three-axis acceleration acting on the upper rotating body 450.

The actual machine output interface 42 includes an actual machine image output device 421 and an actual machine wireless communication device 422. The actual machine image output device 421 is disposed, for example, inside the cab 454 and near the front window. The actual machine image output device 421 may be omitted. In addition, the actual machine wireless communication device 422 performs wireless communication between the work support server 10 and the work machine 40.

The operating mechanism 460 includes a boom 461 that is movably attached to the upper rotating body 450, an arm 463 that is rotatably connected to the tip of the boom 461, and a bucket 465 that is rotatably connected to the tip of the arm 463. The operating mechanism 460 is equipped with a boom cylinder 462, an arm cylinder 464, and a bucket cylinder 466 that are configured as extendable hydraulic cylinders. In addition to the bucket 465, various attachments such as a nibbler, cutter, and magnet may be used as the working unit.

The boom cylinder 462 is interposed between the boom 461 and the upper rotating body 450 so as to extend and retract when supplied with hydraulic oil, thereby rotating the boom 461 in the hoisting direction. The arm cylinder 464 is interposed between the arm 463 and the boom 461 so as to extend and retract when supplied with hydraulic oil, thereby rotating the arm 463 around a horizontal axis relative to the boom 461. The bucket cylinder 466 is interposed between the bucket 465 and the arm 463 so as to extend and retract when supplied with hydraulic oil, thereby rotating the bucket 465 around a horizontal axis relative to the arm 463.

The work support processing of the work support system, which is achieved through the cooperation of the work support server 10, the remote control device 20, and the work machine 40, will be explained using the flowchart shown in FIG. 4.

The work assistance system starts the work assistance process when the remote control device 20 is started. For example, when the operator turns off the cutoff lever 2113, the remote control device 20 transmits an operation signal to the work assistance server 10 (STEP 201).

When the work support server 10 receives the operation signal, it requests the work machine 40, which is the target of operation of the remote control device 20, to send an image (STEP 101).

When the work machine 40 receives the image transmission request, it transmits the images captured by the first imaging device 412a, 412b and the second imaging device 412c (the first captured image and the second captured image) to the work support server 10 (STEP 401).

When the first support processing element 101 of the work support server 10 receives the images (first image, second image) captured by the first imaging device 412a, 412b and the second imaging device 412c, the first support processing element 101 recognizes the forward or backward direction of the lower running body 430 based on the appearance of the first marker M1, which is located in front of the pair of crawlers 430a, 430b and has a different shape from the first marker M1, which is located in front of the pair of crawlers 430a, 430b and is unevenly distributed in the forward and backward directions of each of the pair of crawlers 430a, 430b in the first image captured by the first imaging device 412a, 412b, and the second marker M2, which is located behind the pair of crawlers 430a, 430b (forward and backward direction estimation process) (STEP 102).

The second support processing element 102 generates an indicator image representing the forward or backward direction of the undercarriage 430 recognized by the first support processing element 101 (STEP 103).

The second support processing element 102 also generates a composite image in which the index image is at least partially superimposed on the work machine 40 or its surroundings in the first captured image (STEP 104), and transmits the composite image to the remote control device 20 (STEP 105).

For example, the second support processing element 102 generates a composite image by superimposing the index image I1 on the first captured image (hereinafter also referred to as the foot image) of Figures 6B to 10B captured by the foot camera 412a. The second support processing element 102 also generates a composite image by superimposing the index image I1 on the first captured image (hereinafter also referred to as the left image) of Figure 10C captured by the left camera 412b. The second support processing element 102 also generates a composite image by superimposing the index image I2 on the second captured image (hereinafter also referred to as the main image) of Figures 6C to 9C and Figure 10D captured by the main camera 412c.

In other words, the second support processing element 102 generates a composite image in which the indicator images I1 and I2 are at least partially superimposed on the work machine 40 or its surroundings on at least one of the first captured image (foot image, left image) and the main image at an angle corresponding to the rotation angle of the upper rotating body 450 relative to the lower running body 430 recognized by the first support processing element 101.

Specifically, the first support processing element 101 recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 based on the inclination angle of the left and right crawlers 430a, 430b relative to the imaging direction of the first imaging devices (foot camera, left camera) 412a, 412b and the first marker M1 and second marker M2 shown in the first imaging image (foot image, left image). The second support processing element can generate a composite image by superimposing the index images I1, I2 corresponding to the inclination angle of the left and right crawlers 430a, 430b relative to the imaging direction of the first imaging devices (foot camera, left camera) 412a, 412b, for example.

The remote control device 20 displays the received composite image, for example, on the central remote image output device 2210 (STEP 202). Note that the composite image may be displayed on an image output device other than the central remote image output device 2210, for example, on the left remote image output device 2211, the right remote image output device 2212, etc.

We will now explain the forward/reverse direction estimation process that recognizes the forward or backward direction of the lower running structure 430 based on the appearance of the first marker or the second marker in the first captured image in STEP 102.

As shown in FIG. 5, the first support processing element 101 determines whether the first marker M1 or the second marker M2 is shown in the foot image (STEP 211).

The determination in STEP 211 is made, for example, by referring to the image data (design data) of the first marker M1 and the image data (design data) of the second marker M2 pre-recorded in the database 110, based on whether or not the foot image contains a design that matches the image data (design data) of the first marker M1 or the image data (design data) of the second marker M2.

Specifically, the first support processing element 101 corrects the design of the first marker M1 (or the design of the second marker M2) reflected in the foot image using the aspect ratio, and then calculates the similarity with the image data of the first marker M1 (or the second marker M2) recorded in the database 110. If the similarity is equal to or greater than a predetermined value, the first support processing element 101 determines that the first marker M1 (or the second marker M2) is included in the foot image.

The determination in STEP 211 is not limited to this determination method, and for example, multiple image data of the first marker M1 and the second marker M2 of the foot image according to the angle of the upper rotating body 450 relative to the lower running body 430, and multiple image data of the first marker M1 and the second marker M2 of the left image are recorded in advance in the database 110. The first support processing element 101 may compare the multiple image data with the image data of the foot image and the left image transmitted from the work machine 40 to determine whether the first marker M1 and the second marker M2 are included in the foot image or the left image.

When the first support processing element 101 determines in STEP 211 that the first marker M1 or the second marker M2 is shown in the foot image (STEP 211: YES), it determines whether the marker is the first marker M1 (STEP 212).

The determination in STEP 212 is made, for example, based on the image that the first support processing element 101 determined to be a match in the determination in STEP 211.

When the first support processing element 101 determines in STEP 212 that the marker shown in the foot image is the first marker M1 (STEP 212: YES), it determines whether the number of first markers M1 shown in the foot image is one (STEP 213).

The determination in STEP 213 is made, for example, based on the number of images that the first support processing element 101 determined to be a match in the determination in STEP 211.

When the first support processing element 101 determines that the number of first markers M1 displayed in the foot image is not one (STEP 213: NO), it recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 as 0±α° (STEP 214). In other words, the first support processing element 101 determines that the forward direction in which the front window of the cab 454 is provided coincides with the forward direction of the lower running body 430.

Here, α is the range of angles around 0° within which the two first markers M1 are captured in the foot image. The angle α can be set appropriately depending on the angle of view of the lens used in the first imaging device 412a (wide-angle lens, fisheye lens, etc.), the separation distance between the pair of crawlers 430a, 430b of the lower running body 430, etc. The first support processing element 101 recognizes the backward direction of the lower running body 430 as the direction opposite to the forward direction.

For example, as shown in FIG. 6A, when the forward direction of the lower traveling body 430 of the work machine 40 and the front direction of the upper rotating body 450 are the same, the pair of left and right crawlers 430a, 430b and the two first markers M1 provided on each crawler 430a, 430b are reflected in the foot image shown in FIG. 6B. Therefore, in such a case where the two first markers M1 are reflected in the foot image, the first support processing element 101 determines that the front direction where the front window of the cab 454 is provided and the forward direction of the lower traveling body 430 are the same. Note that, as explained in the synthetic image generation theory of STEP 104 above, the second support processing element 102 superimposes the index image I1 according to the angle on the foot image as shown in FIG. 6B, and superimposes the index image I2 on the main image as shown in FIG. 6C.

That is, the first support processing element 101 recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 based on the reflection manner in the foot image of the pair of crawlers 430a, 430b, which changes in response to the rotation of the upper rotating body 450 relative to the lower running body 430, and the reflection manner in the foot image of each of the first marker M1 and the second marker M2. Note that the first support processing element 101 may recognize the rotation angle by referring to the rotation data acquired by the rotation angle sensor of the actual machine status sensor group 414.

When the first support processing element 101 determines that the number of first markers displayed in the foot image is one (STEP 213: YES), it determines whether the crawler to which the first marker displayed in the foot image is attached is the right crawler 430a (STEP 215).

For example, the first support processing element 101 pre-records in the database 110 a plurality of image data of the right crawler 430a and the left crawler 430b in the foot image according to the angle of the upper rotating body 450 relative to the lower running body 430, and a plurality of image data of the right crawler 430a and the left crawler 430b in the left image. The first support processing element 101 compares the plurality of image data with the image data of the foot image and the left image transmitted from the work machine 40 to determine whether the right crawler 430a is included in the foot image or the left image.

When the first support processing element 101 determines that the crawler shown in the foot image is the right crawler 430a (STEP 215: YES), it recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 as 45±α° (STEP 216). In other words, the first support processing element 101 determines that the angle that the front direction of the cab 454 makes with the forward direction of the lower running body 430 is 45±α°. That is, the first support processing element 101 recognizes that the forward direction of the lower running body 430 is 45° relative to the front direction of the cab 454.

For example, as shown in FIG. 7A, when the front direction of the upper rotating body 450 forms an angle of 45±α° with respect to the forward direction of the lower traveling body 430, the foot image shown in FIG. 7B shows, for example, the right crawler 430a and one first marker M1 provided on the right crawler 430a. Therefore, in such a case where the right crawler 430a and one first marker M1 are reflected in the foot image, the first support processing element 101 determines that the front direction where the front window of the cab 454 is provided forms an angle of 45±α° with respect to the forward direction of the lower traveling body 430. Note that, as explained in the synthetic image generation theory of STEP 104 above, the second support processing element 102 superimposes an index image I1 according to the angle on the foot image as shown in FIG. 7B, and superimposes an index image I2 on the main image as shown in FIG. 7C.

When the first support processing element 101 determines that the crawler shown in the foot image is the left crawler 430b (STEP 215: NO), it recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 as 315±α° (STEP 217). In other words, the first support processing element 101 determines that the angle that the front direction of the cab 454 makes with the forward direction of the lower running body 430 is 315±α°. That is, the first support processing element 101 recognizes that the forward direction of the lower running body 430 is a direction of 315±α° relative to the front direction of the cab 454.

For example, as shown in FIG. 8A, when the front direction of the upper rotating body 450 forms an angle of 315±α° with respect to the forward direction of the lower traveling body 430, the left crawler 430b and one first marker M1 provided on the left crawler 430b are reflected in the foot image shown in FIG. 8B. Therefore, in such a case where the left crawler 430b and one first marker M1 are reflected in the foot image, the first support processing element 101 determines that the front direction where the front window of the cab 454 is provided forms an angle of 315° with respect to the forward direction of the lower traveling body 430. Note that, as explained in the synthetic image generation theory of STEP 104 above, the second support processing element 102 superimposes an index image I1 according to the angle on the foot image as shown in FIG. 8B, and superimposes an index image I2 on the main image as shown in FIG. 8C.

When the first support processing element 101 determines in STEP 212 that the marker shown in the foot image is the second marker (STEP 212: NO), it determines whether the number of second markers M2 shown in the foot image is one (STEP 218).

When the first support processing element 101 determines that the number of second markers M2 displayed in the foot image is not one (STEP 218: NO), it recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 as 180±α° (STEP 219). In other words, the first support processing element 101 determines that the forward direction in which the front window of the cab 454 is provided and the forward direction of the lower running body 430 are in an opposite relationship.

When the first support processing element 101 determines that the number of second markers M2 displayed in the foot image is one (STEP 218: YES), it determines whether the crawler to which the second marker M2 displayed in the foot image is attached is the right crawler 430a (STEP 220).

When the first support processing element 101 determines that the crawler shown in the foot image is the right crawler 430a (STEP 220: YES), it recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 as 135±α° (STEP 221). In other words, the first support processing element 101 determines that the angle that the front direction of the cab 454 makes with the forward direction of the lower running body 430 is 135±α°. That is, the first support processing element 101 recognizes that the forward direction of the lower running body 430 is a direction of 135±α° relative to the front direction of the cab 454.

When the first support processing element 101 determines that the crawler shown in the foot image is the left crawler 430b (STEP 220: NO), it recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 as 225±α° (STEP 222). In other words, the first support processing element 101 determines that the angle that the front direction of the cab 454 makes with the forward direction of the lower running body 430 is 225±α°. That is, the first support processing element 101 recognizes that the forward direction of the lower running body 430 is a direction of 225±α° relative to the front direction of the cab 454.

For example, as shown in FIG. 9A, when the front direction of the upper rotating body 450 forms an angle of 225±α° with respect to the forward direction of the lower traveling body 430, the left crawler 430b and one second marker M2 provided on the left crawler 430b are reflected in the foot image shown in FIG. 9B. Therefore, in such a case where the left crawler 430b and one second marker M2 are reflected in the foot image, the first support processing element 101 determines that the front direction where the front window of the cab 454 is provided forms an angle of 225±α° with respect to the forward direction of the lower traveling body 430. Note that, as explained in the synthetic image generation theory of STEP 104 above, the second support processing element 102 superimposes an index image I1 according to the angle on the foot image as shown in FIG. 9B, and superimposes an index image I2 on the main image as shown in FIG. 9C.

When the first support processing element 101 determines in STEP 211 that the first marker M1 or the second marker M2 is not shown in the feet image (STEP 211: NO), since the first marker M1 or the second marker M2 is shown in either the feet image or the left image as described above, the first support processing element 101 refers to the left image and determines whether the marker shown in the left image is the first marker M1 (STEP 223).

When the first support processing element 101 determines that the marker shown in the left image is the first marker M1 (STEP 223: Y), it recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 as 90±α° (STEP 224). In other words, the first support processing element 101 determines that the angle that the front direction of the cab 454 makes with the forward direction of the lower running body 430 is 90±α°. That is, the first support processing element 101 recognizes that the direction of the lower running body 430 is 90±α° relative to the front direction of the cab 454 as the forward direction.

When the first support processing element 101 determines that the marker shown in the left image is the second marker M2 (STEP 223: NO), it recognizes the rotation angle of the upper rotating body 450 relative to the lower running body 430 as 270±α° (STEP 225). In other words, the first support processing element 101 determines that the angle that the front direction of the cab 454 makes with the forward direction of the lower running body 430 is 270±α°. That is, the first support processing element 101 recognizes that the direction of 270±α° of the lower running body 430 is the forward direction relative to the front direction of the cab 454.

For example, as shown in FIG. 10A, when the front direction of the upper rotating body 450 forms an angle of 270±α° with respect to the forward direction of the lower traveling body 430, for example, only the left crawler 430b is reflected in the foot image shown in FIG. 10B. Therefore, the first support processing element 101 refers to the left image shown in FIG. 10C and recognizes that, for example, a pair of left and right crawlers 430a, 430b and two first markers M1 provided on each crawler 430a, 430b are reflected. Therefore, in such a case where two first markers M1 are reflected in the left image, the first support processing element 101 determines that the front direction where the front window of the cab 454 is provided is at an angle of 270±α° with respect to the forward direction of the lower traveling body 430. As explained in the composite image generation theory of STEP 104 above, the second support processing element 102 superimposes an index image I1 corresponding to the angle on the left image as shown in FIG. 10C, and superimposes an index image I2 on the main image as shown in FIG. 10D.

Therefore, the first support processing element 101 recognizes the forward or backward direction of the lower running body 430 based on the appearance of the first marker M1 and the second marker M2 in the multiple first captured images (foot image, left image) acquired through the multiple first imaging devices 412a, 412b.

In the above description, the types of the first markers M1 and second markers M2 reflected in the foot image and the left image and the number of the first markers M1 and second markers M2 reflected in the foot image and the left image are determined individually in STEPs 211 to 213, 218, and 223. However, these determinations may be made in one step by the image processing described above.

In the above explanation, the rotation angle of the upper rotating body 450 relative to the lower running body 430 was recognized taking into account the type of crawler 430a, 430b reflected in the first captured image (foot image, left image), but this is not limited to this.

For example, the first support processing element 101 may recognize the rotation angle of the upper rotating body 450 relative to the lower running body 430 based on the type of the first marker M1 and the second marker M2 reflected in the first captured image (foot image, left image) and the number of the first marker M1 and the second marker M2 reflected in the first captured image (foot image, left image).

Specifically, the first support processing element 101 recognizes whether the marker reflected in the first captured image is the first marker M1 or the second marker, and recognizes the number of the recognized first markers M1 or second markers reflected in the first captured image.

For example, when the angle at which the front direction of the cab 454 and the forward direction of the lower running body 430 coincide is 0±α°, if two first markers M1 or two second markers M2 are reflected in the first captured image (foot image), the first support processing element 101 recognizes that the rotation angle of the upper rotating body 450 relative to the lower running body 430 is a specific angle, for example, 0° or an angle close to 180° (0±α° or 180±α°).

Furthermore, the first support processing element 101 recognizes that, for example, when one first marker M1 or one second marker M2 is reflected in the first captured image (foot image), the rotation angle of the upper rotating body 450 relative to the lower running body 430 is a specific angle, for example, an angle close to 45° or 135° (45±α° or 135±α°).

Furthermore, the first support processing element 101 can recognize the forward and backward directions of the pair of crawlers 430a, 430b based on the type of marker (first marker M1 or second marker M2) reflected in the first captured image (foot image). Furthermore, if the first marker M1 and the second marker M2 are not reflected in the first captured image (foot image), it is recognized that the rotation angle of the upper rotating body 450 relative to the lower running body 430 is a specific angle, for example, an angle close to 90° (90±α°). Therefore, it is possible to recognize the rotation angle of the upper rotating body 450 relative to the lower running body 430 and the forward and backward directions of the pair of crawlers 430a, 430b without using the reflection mode of the pair of crawlers 430a, 430b in the first captured image (foot image, left image). This eliminates the need for image analysis processing required for the reflection pattern of the pair of crawlers 430a, 430b, thereby reducing the processing load on the second support processing element 102.

The first support processing element 101 may also recognize the rotation angle of the upper rotating body 450 relative to the lower running body 430 based on the left-right positions of the first marker M1 and the second marker M2 in the first captured image (foot image, left image) as the reflection mode of the first marker M1 and the second marker M2 contained in the first captured image (foot image, left image).

According to this aspect, for example, when the angle at which the front direction of the cab 454 and the forward direction of the lower running body 430 coincide is set to 0°, if one of the two first markers M1 (or the two second markers M2) is located to the right of the center of the first captured image (foot image, left image) and the other first marker M1 (or the second marker) is located to the left of the center of the first captured image (foot image), the first support processing element 101 recognizes that the angle of the upper rotating body 450 relative to the lower running body 430 is a specific angle, for example, an angle close to 0° or 180° (0±α° or 180±α°).

The first support processing element 101 may perform a more detailed angle determination, for example, at the position of the midpoint of the line connecting the first markers M1. For example, when the angle of the upper rotating body 450 relative to the lower running body 430 is 3 degrees, the position of the midpoint of the line connecting the first markers M1 is located to the right in the left-right direction of the foot image, compared to when the angle is 0°. Multiple positions of the midpoints of the lines connecting the first markers M1 corresponding to such various angles are recorded in the database 110 as image data. The first support processing element 101 may perform a detailed determination of the angle by referring to the position of the midpoint and calculating the degree of agreement of the position of the midpoint of the foot image.

For example, when the first marker M1 of the right crawler 430a is located near the center in the left-right direction in the first captured image (foot image), the rotation angle of the upper rotating body 450 relative to the lower running body 430 is recognized to be a specific angle, for example, an angle close to 45° (45±α°).For example, when the first marker M1 of the left crawler 430b is located near the center in the left-right direction in the first captured image (foot image), the rotation angle of the upper rotating body 450 relative to the lower running body 430 is recognized to be a specific angle, for example, an angle close to 135° (135±α°).

In this way, the appearance of the first marker M1 and the second marker M2 included in the first captured image changes depending on the rotation angle of the upper rotating body 450 relative to the lower running body 430. Specifically, the left-right positions of the first marker M1 and the second marker M2 in the first captured image change depending on the change in the rotation angle. Therefore, the first support processing element 101 can recognize the rotation angle of the upper rotating body 450 relative to the lower running body 430 by recognizing the left-right positions of the first marker M1 and the second marker M2 in the first captured image (foot image).

The second support processing element 102 may display or hide the composite image based on a predetermined timing. The control of displaying or hiding the composite image by the second support processing element 102 is described below.

As shown in FIG. 11, the remote control device 20 starts the work support process when the remote control device 20 is started, which is triggered by the activation of the remote control device 20. When the operator OP1 turns off the cutoff lever 2113, for example (STEP 231: YES), the remote control device 20 transmits an operation signal to the work support server 10 (STEP 232).

When the first support processing element 101 of the work support server 10 receives the operation signal, it recognizes that the cutoff lever 2113 has been turned OFF (STEP 111: YES), and the first support processing element 101 and the second support processing element 102 perform display processing of the composite image (STEP 112).

The first support processing element 101 therefore acquires operation acceptance information, which is information indicating whether or not operation of the work machine 40 can be accepted in the remote control device 20, as an operation signal. In the display processing of STEP 112, the process from the forward/rearward direction estimation process of STEP 102 to the composite image transmission process of STEP 105 is carried out. Therefore, the second support processing element 102 transmits a composite image to the remote control device 20 when the operation acceptance information indicates that operation of the work machine 40 can be accepted.

The first support processing element 101 of the work support server 10 determines whether a predetermined time (e.g., 60 seconds) has elapsed since the composite image display process of STEP 112 was performed, or whether a previously designated operation (e.g., operation of the boom operation device, arm operation device, or bucket operation device, or a traveling motion described below) has been performed (STEP 113).

When the first support processing element 101 of the work support server 10 determines in STEP 113 that a predetermined time has elapsed since the composite image display process was performed or a pre-specified designated operation has been performed (STEP 113: YES), the first support processing element 101 and the second support processing element 102 perform a process to hide the composite image (STEP 114). That is, the first support processing element 101 and the second support processing element 102 stop the composite image generation process in STEP 104 and the composite image transmission process in STEP 105.

When the first support processing element 101 of the work support server 10 determines in STEP 113 that a predetermined time has elapsed since the composite image display process was performed or that a pre-specified operation has not been performed (STEP 113: NO), the first support processing element 101 proceeds to the determination in STEP 119 described below.

After transmitting an operation signal to the work support server 10 in STEP 232, the remote control device 20 then transmits the third captured image captured by the third imaging device 212 to the work support server 10 (STEP 233).

After performing the non-display process of the composite image in STEP 114, the first support processing element 101 of the work support server 10 refers to the acquired third captured image and determines whether or not the operator OP1 has performed a running motion, which is an action for driving the work machine 40 (STEP 115).

The traveling motion is the motion of the operator OP1 to operate the traveling operation device of the remote operation mechanism 211. Examples of the traveling motion include the motion to operate the operation lever (travel lever) of the traveling operation device and the motion to operate the traveling pedal.

The determination in STEP 115 is made, for example, when the state of the operator OP1 and the travel lever 2110 has transitioned from a non-interaction state to an interaction state. Here, the non-interaction state is, for example, a state in which the operator OP1 is not gripping or touching the travel lever 2110. Also, the interaction state is, for example, a state in which the operator OP1 is gripping or touching the travel lever 2110. Note that the non-interaction state may be a state in which the operator OP1's foot is not on the travel pedal, and the interaction state may be a state in which the operator OP1's foot is on the travel pedal. In this way, it is possible to determine whether or not a travel motion has been performed in an early operation stage in which the lower travel body 430 has not yet started to operate in a situation in which it is highly likely that the operator OP1 intends to operate the lower travel body 430.

Therefore, the first support processing element 101 acquires, as a traveling motion, the traveling operation information of the preparatory operation, which is an operation that indicates the intention to input an operation to move the work machine 40 forward or backward in the remote control device 20.

If the first support processing element 101 determines in the judgment of STEP 115 that the operator OP1 is not performing a running motion, which is an action for running the work machine 40 (STEP 115: NO), it returns to the judgment of STEP 113.

When the first support processing element 101 determines in STEP 115 that the operator OP1 has performed a running motion, which is an action for running the work machine 40 (STEP 115: YES), it determines the type of forward operation or reverse operation, aligns the indicator image with the forward direction or reverse direction to generate a composite image, and performs display processing of the composite image (STEP 116). In the display processing of the composite image in STEP 116, the same processing as the display processing of the composite image in STEP 112 is performed.

Therefore, when the intention represented by the preparatory movement is to input a forward operation, the second support processing element 102 transmits to the remote control device 20 a composite image generated using an indicator image representing the forward direction of the lower running structure 430. When the intention represented by the preparatory movement is to input a reverse operation, the second support processing element 102 transmits to the remote control device 20 a composite image generated using an indicator image representing the reverse direction of the lower running structure 430.

After transmitting the third captured image to the work support server 10 in STEP 233, the remote control device 20 then transmits to the work support server 10 an operation signal corresponding to the operation performed by the operator OP1 to operate the work machine 40 (STEP 234).

The first support processing element 101 determines whether or not a driving operation has been performed based on the operation signal transmitted from the remote control device 20 (STEP 117). The first support processing element 101 performs the determination in STEP 117, for example, based on whether or not the operation signal includes a signal related to a driving operation.

If the first support processing element 101 determines in STEP 117 that driving operations are not being performed (STEP 117: NO), it returns to the determination in STEP 113.

When the first support processing element 101 determines in STEP 117 that a driving operation has been performed (STEP 117: YES), the first support processing element 101 and the second support processing element 102 perform a process of hiding the composite image (STEP 118). In the process of hiding the composite image in STEP 118, the same process as in STEP 114 is performed.

The first support processing element 101 determines whether the shutoff lever 2113 is turned ON (STEP 119).

When the first support processing element 101 determines in STEP 119 that the shutoff lever 2113 is turned ON (STEP 119: YES), it ends the processing.

When the first support processing element 101 determines in STEP 119 that the cutoff lever 2113 is turned OFF (STEP 119: NO), it returns to the determination in STEP 113.

In the above embodiment, the work support process is performed by the work support server 10. However, the work support process may be performed by the remote control device 20 or the work machine 40.

In addition, in the above embodiment, the first support processing element 101 and the second support processing element 102 are described as different processing elements, but each support processing element may be processed by a single "support processing element."

In the above embodiment, the determination of whether the operator OP1 has performed a running motion is performed by referring to the third captured image acquired by the third imaging device 212, but this is not limiting. For example, a grip sensor that detects that the operator has gripped the running lever 2110 may be disposed at the portion where the operator OP1 grips the running lever 2110, and the first support processing element 101 may determine whether the running motion has been performed by acquiring the detection by the grip sensor. The grip sensor may include, for example, a sensor electrode disposed on the running lever 2110 and a measuring unit that measures the capacitance value between the grip sensor and the human body (operator OP1), or may include a measuring unit such as a piezoelectric element that measures the pressure generated when the operator OP1 touches the running lever 2110.

In the above embodiment, if it is determined that a designated operation has been performed (STEP 113: YES), a process of hiding the composite image (STEP 114) is performed. However, STEP 114 does not have to be executed immediately. For example, even if it is determined that a designated operation or a running motion has been performed (STEP 113: YES), STEP 114 may be executed after a predetermined time (e.g., 60 seconds) has elapsed since the display process of the composite image in STEP 112. In this case, the predetermined time can be set arbitrarily, and may be set to be shorter than 60 seconds. This allows the operator OP1 to reliably recognize the forward or backward direction of the lower traveling body 430.

In the above embodiment, when it is determined that a driving operation has been performed (STEP 117: YES), a process to hide the composite image is performed (STEP 118). However, the process of STEP 118 is a process that is executed arbitrarily, and for example, the display process of the composite image may be continued while a driving operation is being performed.

In the above embodiment, the indicator image is superimposed on the first captured image (foot image, left image). However, the indicator image does not necessarily have to be superimposed on the first captured image (foot image, left image). For example, the indicator image may be superimposed only on the second captured image (main image). In other words, the second support processing element 102 may transmit to the remote control device 20 a composite image in which an indicator image representing the forward or backward direction of the lower traveling body 430 recognized by the first support processing element 101 is at least partially superimposed on the work machine 40 or its surroundings on the second captured image acquired through the second imaging device 412c provided separately from the first imaging devices 412a, 412b on the upper rotating body 430.

10... Work support server 101... First support processing element 102... Second support processing element 20... Remote control device 40... Work machine 430... Lower traveling body 430a, 430b... Crawler 450... Upper rotating body

Claims (10)

A remote control system having a mutual communication function between a work machine having a lower traveling body equipped with a pair of left and right crawlers and an upper rotating body that can rotate relative to the lower traveling body, and a remote control device for remotely controlling the work machine,
a first support processing element that recognizes a forward direction or a backward direction of the lower traveling body based on the respective reflection patterns of a first marker provided in front of the pair of crawlers and a second marker provided behind the pair of crawlers, the first marker and the second marker being different in shape and arranged unevenly in the front-rear direction of each of the pair of crawlers in a first captured image acquired through a first imaging device that is provided on the upper rotating body and is provided so as to include at least one of the crawlers of the lower traveling body in an imaging area;
a second support processing element that transmits to the remote operation device a composite image in which an index image representing a forward direction or a backward direction of the undercarriage recognized by the first support processing element is at least partially superimposed on the work machine or its surroundings in the first captured image; and
A remote control system comprising: the first support processing element recognizes a rotation angle of the upper rotating body relative to the lower traveling body based on a reflection manner of the pair of crawlers that changes in response to the rotation of the upper rotating body relative to the lower traveling body and on a reflection manner of each of the first marker and the second marker;
The remote control system according to claim 1, characterized in that the second support processing element transmits to the remote control device a composite image in which the index image is at least partially superimposed on the work machine or its surroundings in the first captured image at an angle corresponding to the rotation angle of the upper rotating body relative to the lower running body recognized by the first support processing element. The remote control system according to claim 2, characterized in that the second support processing element transmits to the remote control device a composite image in which an index image showing the forward or backward direction of the undercarriage according to the rotation angle of the upper rotating body relative to the undercarriage is at least partially superimposed on the work machine or its surroundings in the second captured image acquired through a second imaging device provided on the upper rotating body separately from the first imaging device. The remote control system according to claim 2 or 3, characterized in that the first support processing element recognizes the rotation angle of the upper rotating body relative to the lower traveling body based on the distinction between the first marker and the second marker reflected in the first captured image and the number of the first marker and the second marker reflected in the first captured image. The remote control system according to any one of claims 2 to 4, wherein the first support processing element recognizes a rotation angle of the upper rotating body relative to the lower traveling body based on a left-right position of the first marker or the second marker in the first captured image as a reflection mode of the first marker and the second marker included in the first captured image. The remote operation system according to any one of claims 1 to 5, wherein the first support processing element acquires the first captured image including the first marker formed in a three-dimensional shape protruding from the lower traveling body or the second marker formed in a three-dimensional shape protruding from the lower traveling body. The first imaging device is disposed in a plurality of positions facing different directions from each other in a rotation direction of the upper rotating body relative to the lower traveling body,
The first support processing element recognizes a forward direction or a backward direction of the lower traveling body based on a reflection state of each of the first marker and the second marker in a plurality of first captured images acquired through a plurality of the first imaging devices, and
The second support processing element transmits to the remote control device the composite image obtained by superimposing the index image on the first captured image used by the first support processing element when recognizing the forward direction or the backward direction of the lower traveling body.
7. The remote control system according to claim 1, wherein the remote control system comprises: a first input section; The first support processing element acquires operation acceptance information which is information indicating whether or not an operation of the remote control device for the work machine can be accepted (specific example: ON/OFF of a shutoff lever),
The remote operation system according to claim 1 , wherein the second support processing element transmits the composite image to the remote operation device when the operation acceptance information indicates that an operation of the work machine can be accepted. The first support processing element acquires travel operation information (specific examples: motion of the operator's hands and feet, operation of a travel lever, etc., and touching of the travel lever) of a preparatory operation which is an operation representing an intention to input an operation input for moving the work machine forward or backward in the remote operation device,
The second assistance processing element transmits to the remote control device the composite image generated using the indicator image representing the forward direction of the lower running structure when the intention represented in the preparatory movement is the operation input for forward movement, and transmits to the remote control device the composite image generated using the indicator image representing the backward direction of the lower running structure when the intention represented in the preparatory movement is the operation input for reverse movement. A remote control system having a mutual communication function between a work machine having a lower traveling body equipped with a pair of left and right crawlers and an upper rotating body that can rotate relative to the lower traveling body, and a remote control device for remotely controlling the work machine,
a first support processing element that recognizes a forward direction or a backward direction of the lower traveling body based on the respective reflection patterns of a first marker provided in front of the pair of crawlers and a second marker provided behind the pair of crawlers, the first marker and the second marker being different in shape and arranged unevenly in the front-rear direction of each of the pair of crawlers in a first captured image acquired through a first imaging device that is provided on the upper rotating body and is provided so as to include at least one of the crawlers of the lower traveling body in an imaging area;
a second support processing element that transmits to the remote control device a composite image in which an index image indicating the forward or backward direction of the undercarriage recognized by the first support processing element is at least partially superimposed on the work machine or its surroundings on a second captured image acquired through a second imaging device provided on the upper rotating body separately from the first imaging device; and
A remote control system comprising:

Images