JP7820082B2 - Information Presentation System - Google Patents

Description

The present invention relates to an information presentation system that presents information about the distance to an object during work.

In recent years, many technologies have been developed that allow construction machinery at work sites and other locations to be operated remotely. When performing such remote operation, images taken by a camera attached to the construction machinery are transmitted to a remote location, and the operator remotely controls the construction machinery based on the images displayed there.

In general, construction machinery, apart from moving (driving) around the site, also performs a variety of tasks such as loading and transporting objects using a movable work body (for example, a backhoe bucket) and leveling soil. During such operations, if the operator is sitting in the driver's seat at the actual site, they can visually grasp the distance to the object being worked on (the soil to be transported/loaded in the case of transporting/loading operations, or the soil to be leveled in the case of leveling operations). However, with remote control, the operator sees the site through camera footage, making it difficult for the operator to grasp the actual distance.

Here, there is known prior art in which a stereo camera is attached to construction machinery such as a shovel, the stereo camera detects the distance to an object, and a frame is displayed surrounding the object when the object is closer than a predetermined distance (see, for example, Patent Document 1). This prior art also makes it possible to change the color of the frame depending on the detected distance, so it is somewhat relevant in terms of conveying a sense of distance.

Other prior art techniques are known that attach a camera to a construction machine such as a crane to capture images of the end of the boom (jib), and use the focal point during autofocus photography and the detected length of the boom to detect obstacles that may collide with the boom (see, for example, Patent Document 2). This prior art technique also makes it possible to obtain information that can identify the location of obstacles.

Japanese Patent Application Laid-Open No. 2019-4484 JP 2018-24500 A

The former prior art (Patent Document 1) switches the display depending on whether the distance from the excavator body is less than a predetermined distance, making it easier to recognize features that are relatively close to the excavator body, i.e., features with a relatively high possibility of contact. If this technology were applied to remote operation, it would be possible to convey to a remote operator, to some extent, a sense of distance to surrounding features. However, this technology only controls the display according to the distance between the excavator body and surrounding features, and is unable to convey to the operator a sense of distance between a work vehicle, such as a bucket used for transporting and loading work, and the target object.

Furthermore, the latter prior art (Patent Document 2) only notifies the operator of the presence of an object that may be hit with the current boom length, and does not provide the operator with a sense of distance to the object during operation (while moving the boom, arm, or bucket to approach).

The present invention aims to provide information that allows operators to easily grasp distances when transporting objects, even in remote locations.

The present invention provides an information presentation system. This system measures the distance between a set object and a work unit, and presents information in a format that corresponds to that distance. Therefore, an operator at a remote location can easily grasp the sense of the distance between the work unit and the object from the format of the presented information.

The system of the present invention also takes into account that the distance to an object changes dynamically during the course of work. For example, during work such as transporting, loading, or leveling soil, the distance to an object such as soil changes dynamically depending on the operation of the bucket, which is the work object. For this reason, the manner in which information such as distance is presented changes as the distance between the work object and the object gets closer (or farther). This makes it easier for an operator at a remote location to get a sense of the dynamically changing distance to the object from the changes in the manner of the information presented by the system.

The following techniques can be used to measure distance using the system of the present invention.
That is, the distance (1) from a predetermined reference position to the object is measured, the distance (2) from the predetermined reference position to the working body is measured, and the distance between the working body and the object is measured based on these measured distances (1) and (2). This allows for highly accurate distance measurement and information presentation.

Preferably, the system of the present invention can include a stereo camera. Unlike a movable workpiece, the stereo camera captures images of the workpiece from a predetermined reference position that is constant (unmoving) relative to the target object. The distance from the reference position to the workpiece is then measured based on the image of the workpiece captured by the stereo camera (image features such as contours and edges that can be extracted from the image data). This makes it possible to realize a system using general-purpose (existing) technology.

Alternatively, the system of the present invention includes a camera that captures an image of the work object from a predetermined reference position that is fixed (unmoving) relative to the object, with the actual horizontal length (actual dimension) of the work object set. The distance from the reference position to the work object is then calculated (measured) based on the horizontal length in the image captured by the camera and the set actual length. This makes it possible to realize a system using general-purpose (existing) technology, similar to the above.

In the system of the present invention, the target can be set as follows.
That is, the system includes a camera that captures an image of an object, and the image captured by the camera is displayed on a device that allows touch operation on the screen (such as a touch screen). An operator receiving information from the system can easily specify the object to be worked on by performing a touch operation on the screen. The system can set the object from the image area specified by the touch operation.

The system of the present invention may also include a distance sensor. The distance sensor can measure the distance from a predetermined reference position, and the distance to the object measured by the distance sensor can also be used to measure the distance between the work body and the object. This makes it possible to realize a system using general-purpose (existing) technology, as described above.

The system of the present invention may be capable of switching modes. The system may have at least one mode in which information is presented (presentation mode) and one in which information is not presented (non-presentation mode). For example, when the system is in a mode in which information is not presented (non-presentation mode), if an operation is performed to specify transporting an object as work to be performed by the work object, this triggers the system to switch to the presentation mode. Furthermore, if an operation other than work is performed by the work object while in a mode in which information is presented (presentation mode), this triggers the system to end the presentation mode and switch to the non-presentation mode. This allows the system to refrain from presenting information in situations where a sense of distance is not relevant, for example, when operating at a remote location, and to present information in a timely manner when a need arises to grasp a sense of distance.

The present invention can present information that makes it easier for operators to grasp distances when transporting objects, even in remote locations.

FIG. 1 is a block diagram showing a configuration of an information presentation system 100. FIG. 1 is a side view of a backhoe BH. FIG. 1 is a plan view of a backhoe BH. FIG. 2 is a diagram showing an example of an overhead image generated by the system 100. FIG. 2 is a diagram showing an example of a forward image generated in the system 100. FIG. 10 is a diagram showing a front image at the end of the movement of the backhoe BH. FIG. 10 is a view showing a front image at the start of transport work using the bucket BK. FIG. 10 is a diagram showing a front image of a touch screen TS2 installed in an operation room R. 10A and 10B are diagrams showing information presented on the touch screen TS2 when the bucket BK is relatively far from the object. 10A and 10B are diagrams showing the type of information presented on the touch screen TS2 when the distance between the bucket BK and the object becomes closer. 10 is a flowchart illustrating an example of a procedure for information presentation management processing. 10 is a flowchart illustrating an example of a procedure for measuring the distance between buckets. 10 is a flowchart illustrating an example of a procedure for measuring a distance between objects. 10 is a flowchart illustrating an example of a procedure for remote operation management processing. 10 is a flowchart illustrating an example of a procedure for an operation input process. FIG. 10 is a diagram showing the arrangement of various devices in a second embodiment of the system 100.

Embodiments of the present invention will be described below with reference to the drawings. In the following embodiments, an example of an information presentation system is given in which the device side and the remote side are configured separately, but the present invention is not limited to this and may also be a system in which the devices are physically close to each other or integrated.

Figure 1 is a block diagram showing the configuration of the information presentation system 100. In addition to the block configuration of the system 100, Figure 1 also shows a construction site where construction machinery (e.g., a backhoe BH) to which the system 100 is applied is located, and an operation room R where the machinery is remotely operated. Note that while a backhoe BH is an example of construction machinery, the system 100 of this embodiment can also be applied to other machinery, such as bulldozers.

As described above, in this embodiment, the backhoe BH deployed on-site is operated from an operation room R located in a remote location, so the configuration of the system 100 is broadly divided into an "equipment side" and a "remote side." Therefore, the configuration of the "equipment side" will be explained first.

[Equipment side]
The equipment configuration of the system 100 is mounted on a backhoe BH. While the specific layout will be described later using another drawing, the backhoe BH is equipped with a distance sensor 102 mounted on its body, as well as four fisheye cameras 104, 106, 108, and 110. The distance sensor 102 can be a three-dimensional area sensor, such as an existing laser sensor or ultrasonic sensor, and can detect the distance to various objects within the sensing area. Similarly, existing products can be used for the fisheye cameras 104-110. Images captured by the fisheye cameras 104-110 are synthesized (including stereo processing and bird's-eye image processing) and transmitted to a remote location as needed. The synthesized images will be described further below.

The system 100 includes an equipment control unit 120. Detection signals from the distance sensor 102 and image capture signals from the fisheye cameras 104-110 are input to the equipment control unit 120 via an interface (not shown). The equipment control unit 120 is a computer device including, for example, a central processing unit (CPU) (not shown) and its peripheral devices, and can be implemented using an existing laptop PC or the like. The equipment control unit 120 includes various functional blocks, such as a detection processing unit 122, a bucket-to-bucket distance measurement unit 124, an object setting unit 126, an object-to-object distance measurement unit 128, a bucket-to-object distance measurement unit 130, and a presentation information processing unit 132. These functional blocks can be implemented by software processing using, for example, application software, and therefore do not require separate hardware. An overview of the processing performed by these functional blocks is provided below, but specific processing details will be described later using separate flowcharts.

The detection processing unit 122 performs processing to detect (extract pixels from) buckets BK from images captured by, for example, the fisheye cameras 104 and 106. The processing to detect buckets BK from images can be performed, for example, by image recognition using image features of the buckets BK.

The bucket distance measurement unit 124 uses stereo processing to detect the distance from a predetermined reference position to the bucket BK based on the detection results of the detection processing unit 122. The fisheye cameras 104 and 106 that capture images of the bucket BK are attached to the front side of the backhoe BH, and their imaging range covers the working range of the bucket BK. Note that the reference position on the backhoe BH and the positional relationship with the fisheye cameras 104-110 are assumed to have been initially set (calibrated) in advance.

The object setting unit 126 sets the object (e.g., earthwork) to be transported by the bucket BK. The object can be set via communication with a remote site; for example, when an operator at a remote location (operation room R) specifies the object on the screen of the touch screen TS2, the specified pixel area is transmitted to the equipment control unit 120. Note that the specification of the object and the setting of the object on the remote site will be discussed in more detail below.

The object distance measurement unit 128 performs processing to detect the distance to the object. Once the object is set by the object setting unit 126, the object distance measurement unit 128 performs processing to measure the distance from a predetermined reference position to the object, for example, by stereo processing using images captured by the fisheye cameras 104, 106. Alternatively, the object distance measurement unit 128 can measure the distance from a predetermined reference position to the object using a detection signal from the distance sensor 102. Note that the reference position on the backhoe BH and its positional relationship with the distance sensor 102 are also assumed to have been initially set (calibrated) in advance.

The bucket-to-object distance measurement unit 130 performs processing to measure the distance between the bucket BK and the object. In this processing, the distance Db-o between the bucket BK and the object is calculated using three-dimensional processing or the like based on the distance Db to the bucket BK measured by the bucket distance measurement unit 124 and the distance Do to the object measured by the object distance measurement unit 128.

The presentation information processing unit 132 performs processing to synthesize the images captured by the fisheye cameras 104-110 as described above and generate an overhead image. In addition to the overhead image, the presentation information processing unit 132 also performs processing to generate a forward stereo image that includes the bucket BK and the object.

The presentation information processing unit 132 also performs the following processing, which is characteristic of the system 100 of this embodiment. That is, the presentation information processing unit 132 generates information to be presented at the remote location according to the distance calculated (measured) by the bucket-object distance measurement unit 130. The generated information is presented in a manner that conveys a sense of the distance between the bucket BK and the object to the operator at the remote location. Examples of presentation modes that convey a sense of distance include the following (however, they are not limited to the following examples):
(1) The distance between the bucket BK and the object is communicated by voice (e.g., "It's 1.5 meters").
(2) The drawing color on the screen changes depending on the distance (for example, the closer the object, the darker the color).
(3) The output of the alarm sound is changed depending on the distance (for example, the closer the distance, the higher the sound is output).
(4) The alarm sound cycle is changed depending on the distance (for example, the closer the distance, the shorter the cycle of the sound).

The device control unit 120 also includes a memory unit 136 and a communication processing unit 134. The memory unit 136 is implemented using a storage medium such as a ROM, RAM, or HDD. The communication processing unit 134 is a functional block that performs wireless or wired data communication, and is implemented using a unit that is compatible with the communication protocol used by the system 100. The communication processing unit 134 may be a separate component that is physically separated from the device control unit 120.

The equipment control unit 120 also executes control processing for the drive unit 138. The drive unit 138 is configured to include various actuators (hydraulic actuators, internal combustion engines, and their control units, etc.) possessed by the backhoe BH. The equipment control unit 120 executes control processing for the drive unit 138, enabling remote operation of the backhoe BH.

[Remote side]
Next, the configuration on the remote side will be described.
The remote side of the system 100 is installed in an operation room R. That is, various devices such as an operation unit 140, a sound output unit 142, a display unit 144, and a touch input unit 146 are installed in the operation room R. The operation unit 140 is a physical interface that accepts operation inputs from an operator and can be realized, for example, by a remote control controller CN. The sound output unit 142 is, for example, an audio device such as a speaker or a headset (not shown). The display unit 144 and the touch input unit 146 are realized, for example, by a touch-operated display device. In this embodiment, two touch screens TS1 and TS2 are installed in the operation room R as examples of the display unit 144 and the touch input unit 146. The operator can perform touch operations (touch inputs) by touching the screens of the touch screens TS1 and TS2.

The system 100 includes a remote control unit 150. Operation signals from the operation unit 140 and touch operation signals from the touch input unit 146 are input to the remote control unit 150 via an interface (not shown). The remote control unit 150 also outputs audio signals to the sound output unit 142 and image signals to the display unit 144. These audio signals and image signals are output via an external output interface (not shown). The remote control unit 150 is also a computer device including, for example, a CPU (central processing unit) (not shown) and its peripheral devices, and can be implemented using an existing laptop PC or the like. The remote control unit 150 includes functional blocks such as a remote information processing unit 152, and such functional blocks can be implemented by software processing using, for example, application software.

The remote information processing unit 152 executes processing to generate remote operation information, area designation information, etc. to be transmitted to the remote side of the system 100 based on operation signals input from the operation unit 140 and touch operation signals input from the touch input unit 146. The remote control unit 150 also includes a communication processing unit 154. The communication processing unit 154 is also a functional block that performs wireless or wired data communication, and is realized using a unit that is compatible with the communication protocol used by the system 100. Note that the communication processing unit 154 may be physically separated from the remote control unit 150.

In the system 100, the equipment control unit 120 and the remote control unit 150 can communicate data with each other via the communication network 160. The communication network 160 can be an existing network compliant with various communication methods, and can be either a wired or wireless communication method. This makes it possible to remotely control the backhoe BH from the operation room R. Furthermore, information on the distance on the equipment side measured by the equipment control unit 120 can be transmitted to the remote control unit 150, and can be presented remotely on the sound output unit 142 and display unit 144 in a manner that makes it easy to grasp the sense of distance.

The backhoe BH to which the system 100 is applied has the following configuration.
Fig. 2 is a side view of the backhoe BH, and Fig. 3 is a plan view of the backhoe BH. The backhoe BH is mainly composed of a traveling body MV, a revolving body TR, and a front working mechanism FR.

The traveling body MV rotates, for example, a pair of crawlers CW to move the backhoe BH in a desired direction within the work site.
The rotating unit TR is supported on the upper part of the running unit MV via a rotating device GR, and the rotating device GR rotates the rotating unit TR about a vertical axis relative to the running unit MV. A driver's cab CB is provided in front of the rotating unit TR, and in this driver's cab CB, a seat (not shown) as well as operation levers, operation pedals, etc. are installed. However, in this embodiment, application of system 100 leaves the driver's cab CB basically unmanned.

The front working mechanism FR is equipped with a boom BM, arm AM, bucket BK, and multiple hydraulic actuators HD that operate these. The front working mechanism FR is supported on the rotating unit TR so that it can swing back and forth, and the entire front working mechanism FR changes direction as the rotating unit TR rotates. The operation and movement of the backhoe BH are well known, so a detailed description will be omitted.

Next, we will explain the placement of various devices on the backhoe BH with reference to Figures 2 and 3.

The distance sensor 102 is mounted, for example, at the upper front of the rotating unit TR (on the roof of the driver's cab CB). In this mounting position, the distance sensor 102 mainly detects the distance to objects in the space in front of the rotating unit TR. The predetermined reference position can be set, for example, on the rotating unit TR.

The fish-eye cameras 104 to 110 are arranged as follows:
Two fisheye cameras 104, 106 are attached to the front of the rotating unit TR, at the left and right corners. Of these, the fisheye camera 104 at the front left corner captures wide-angle images (e.g., a field of view of approximately 270°) of an area including the left front and left lateral sides of the rotating unit TR. The fisheye camera 106 at the front right corner captures wide-angle images of an area including the right front and right lateral sides of the rotating unit TR. The imaging ranges of these fisheye cameras 104, 106 cover the working range of the bucket BK. Therefore, when work is performed using the bucket BK, the bucket BK is stereo-imaged by the fisheye cameras 104, 106.

Two fisheye cameras 108, 110 are attached to the rear of the rotating unit TR (counterweight WT), at the left and right corners. Of these, the fisheye camera 108 at the rear left corner captures wide-angle images of the area including the left rear and left lateral sides of the rotating unit TR. The fisheye camera 110 at the rear right corner captures wide-angle images of the area including the right rear and right lateral sides of the rotating unit TR.

[Example of an overhead image]
4 is a diagram showing an example of an overhead image generated by the system 100. The overhead image is generated as an image taken from a virtual viewpoint set above the backhoe BH. In this example, it appears as if the camera viewpoint is directly above the backhoe BH, but in reality, it is a composite image of images taken by the fisheye cameras 104 to 110, and the backhoe BH itself is an icon image.

Such an overhead image is transmitted from the equipment control unit 120 to the remote control unit 150 in the system 100 and displayed on the display unit 144 (touch screen TS1) at a remote location. This allows the operator in the operation room R to view the situation around the backhoe BH from the overhead image. Note that the overhead image may also be generated as if it were captured from a virtual viewpoint diagonally above (a so-called bird's-eye view). Furthermore, since the method for generating such overhead images is publicly known, a detailed explanation thereof will be omitted here.

[Example of forward image]
5 is a diagram showing an example of a forward image generated by the system 100. The forward image is generated as an image captured from a virtual viewpoint in the cab CB, for example. In this example, it appears as if the camera viewpoint is located inside the cab CB of the backhoe BH, but in reality, it is a composite image of images captured by the fisheye cameras 104 and 106.

Such a forward image is also transmitted from the equipment control unit 120 to the remote control unit 150 in the system 100 and displayed on the display unit 144 (touch screen TS2) at a remote location. This allows the operator in the operation room R to view the situation in front of the backhoe BH, including the working range of the bucket BK, from the image. Note that the forward image may also be generated as an image taken from a forward position other than the driver's cab CB. Furthermore, since the method of generating such a forward image is publicly known, a detailed description thereof will be omitted here.

[Information presentation overview]
Next, an overview (general flow) of information presentation by the system 100 will be described.
For example, when the backhoe BH is moving within a work site, the system 100 is in a mode (non-presentation mode) in which information regarding the distance between the bucket BK and the object is not presented. Even in this non-presentation mode, the system 100 transmits information such as the overhead image shown in Fig. 4 and the forward image shown in Fig. 5 to the remote side as needed (in real time).

[When traveling]
For example, suppose the operator decides to make the mound FL displayed on the forward image (FIG. 5) displayed on the display unit 144 the object of the transport work. In addition to the mound FL, the forward image also shows a distant view of the ground ahead, vehicles such as trucks TK present on the premises, and vertical structures BD. Relying on this image (forward image) information, the operator performs an operation to move the entire backhoe BH (vehicle body) closer to the mound FL.

[At the end of movement]
Figure 6 shows a front image when the backhoe BH has finished moving. The operator determines from the front image (Figure 6) displayed on the display unit 144 that the entire backhoe BH (the workable range of the bucket BK) has come close enough to the mound of earth FL to be transported, and stops moving the backhoe BH. At this time, the mound of earth FL is displayed large in the foreground in the image, allowing the operator to recognize that it is within the workable range of the bucket BK. However, even if it is within the workable range of the bucket BK, it is difficult for the operator to get a sense of the specific distance between the bucket BK and the mound of earth FL from the image information alone.

[At the start of transportation work]
7 is a diagram showing a front image at the start of transport work using the bucket BK. For example, when the operator performs an operation to transition to transport work using the operation unit 140 (controller CN), the start of transport work is transmitted from the remote control unit 150 to the equipment control unit 120 in the system 100.
Specifically, when the operator operates the operation unit 140 to lower the bucket BK toward the mound FL, the remote control unit 150 of the system 100 automatically determines this and notifies the equipment control unit 120 of the start of the transport work. At the same time, an operation signal to actually lower the bucket BK is sent from the remote control unit 150 to the equipment control unit 120. Alternatively, the operator may operate the operation unit 140, causing the remote control unit 150 to explicitly send a transport work start signal to the equipment control unit 120.

The equipment control unit 120, which receives the operation signal, outputs a control signal to the drive unit 138 and controls the operation of various actuators. As a result, the bucket BK moves closer to the mound FL, as shown by the two-dot chain line in Figure 7. At this time, the forward image shows not only the moving bucket BK and arm AM, but also a truck TK passing by in the distance. Also, although not shown, in the distance, workers walking or flying objects such as airplanes and birds may be seen.

[Processing on the device side]
The device control unit 120 executes the following process as a trigger when the remote control unit 150 notifies the start of the transport work.

(1) Measuring the Distance to the Bucket First, the inter-bucket distance measurement unit 124 measures the distance from the reference position to the bucket BK. Although known techniques can be used to measure the distance, the following processing is performed in this embodiment.
First step: Using several frames of images captured by the fish-eye camera 104 on the front left side, moving objects present in the images are detected.
Second step: The fish-eye camera 106 on the front right side also detects moving objects in the same way.
Third step: Because the bucket BK is being lowered, the moving objects detected up to this point should include the bucket BK. However, if there is anything other than the bucket BK moving within the screen, it will also be detected as a moving object. In the example of Figure 7, the truck TK in the distance would be detected as a moving object.
Therefore, the likelihood of a moving object being a bucket BK is determined from the image (image feature amount) of the moving object, and a moving object that is a bucket BK is detected from among the images. Specifically, since the lower edge of the bucket BK, which is perpendicular to the arm AM, is usually horizontal, a moving object that includes a horizontal edge EG in the image is determined to be a bucket BK. Here, the edge EG indicated by the two-dot chain line in FIG. 7 is used. Note that a bucket BK may also be determined based on edges, contours, or other image feature amounts other than the horizontal direction in the image.
Fourth step: The left and right ends of the bucket BK are detected from the edge EG of the moving object that has been determined to be the bucket BK.
Fifth step: Calculate the distance Db from the reference position to the bucket BK by stereo processing. This calculation can be performed because the bucket BK is captured by the two fisheye cameras 104 and 106. As described above, the positional relationship between the reference position and the fisheye cameras 104 to 110 has been initially set (calibrated) in advance.

(2) Measuring the Distance to the Object Figure 8 shows the forward image of the touch screen TS2 installed in the operation room R. An operator at a remote location can specify the area of the object by touching the area of the mound FL on the screen of the touch screen TS2. When a touch operation is performed, a frame image (shown by a dashed line) surrounding the object is displayed on the screen. Note that by providing a certain degree of redundancy in the device's sensitivity to touch operations, it is possible to absorb slight deviations between the operator's intended object and the touch position on the screen.

When the operator performs a touch operation, the remote control unit 150 transmits the area (pixel area) of the object on the image to the device control unit 120. In response, the object setting unit 126 sets the specified area in the stereo image as the object, and the object distance measurement unit 128 measures the distance Do from the reference position to the object. Typically, an area contains multiple different points (pixels), so the results measured from the stereo image also contain distances to multiple different points. For this reason, the distance to the nearest point is measured here as the distance Do to the object. Note that for convenience, the distance to the nearest point is referred to as the distance to the object, but this is not limited to this. The distance Do to the object may also be measured using the distance sensor 102 as described above.

(3) Calculation of the distance between the bucket and the object Using the distances Db and Do measured in (1) and (2) above, the bucket-object distance measurement unit 130 calculates the distance Db-o between the bucket BK and the object by three-dimensional processing.

(4) Presentation process: Information such as images and sounds that convey a sense of distance to the operator at a remote location is generated and output according to the distance Db-o calculated in (3) above. Specific examples of how information is presented have already been described, but one example is shown below.

[At long distance]
FIG. 9 is a diagram showing information presented on the touch screen TS2 when the distance between the bucket BK and the object is relatively far. In this example, a pop-up window is displayed on the screen, displaying the text information "object-bucket distance" along with distance information of "1.25 M." This distance information corresponds to the distance Db-o measured by the remote control unit 150 (bucket-object distance measuring unit 130). The frame of the pop-up window is displayed in a light color, blue, or the like, depending on the distance b-o. Additionally, the sound output unit 142 may output audio guidance such as "It's 1.25 meters," or an alarm sound (such as "beep beep beep").

[Mid-distance]
Although not specifically illustrated, when the distance to the object decreases to a moderate level during the process of lowering the bucket BK, the type of information presented on the touch screen TS2 changes. For example, a pop-up window continues to be displayed on the screen, displaying the distance information "0.95 m" along with the text information "object-bucket distance." The frame of the pop-up window is displayed in a medium-dark color, yellow, or the like depending on the distance Db-o at that time. The sound output unit 142 may also output audio guidance such as "0.95 meters" or an alarm sound (such as a medium-frequency "beep beep beep").

[At close range]
10 is a diagram showing information presented on the touch screen TS2 when the distance between the bucket BK and the object becomes closer. In this example, a pop-up window is displayed on the screen, displaying the text information "object-bucket distance" along with the distance information "0.63 M." The frame of the pop-up window is displayed in a color such as dark or red depending on the distance Db-o. Similarly, the sound output unit 142 may output audio guidance such as "It's 0.63 meters" or an alarm sound (such as a short-cycle "beep beep beep...!").

Note that presenting information in this manner is useful when lowering the bucket BK toward an object or loading an object into the bucket BK, but is not useful when raising the bucket BK or rotating the rotating unit TR. For this reason, the system 100 of this embodiment can be set to a normal display (non-presentation mode) that does not present information conveying distance when not transporting or loading an object.

[Remote processing]
As previously mentioned, the remote control unit 150 performs the following processes.
Under normal circumstances, the image data transmitted from the equipment control unit 120 is reproduced and processed, and an overhead image and a forward image are displayed on the display unit 144 (touch screens TS1 and TS2). The operator remotely controls the backhoe BH using the operation unit 140 while viewing these images.

[During transport operations]
When carrying out a transport operation, the operator starts the operation of lowering the bucket BK onto the object. At this time, the remote control unit 150 automatically determines that the mode is transport mode based on the operation on the operation unit 140, and notifies the equipment control unit 120 that the transport operation has started. In addition, it sends a signal to the equipment control unit 120 that the operation of lowering the bucket BK has started.
The operator designates the location of the object to be transported by touching the screen of the touch screen TS2 (see FIG. 8). The remote control unit 150 notifies the equipment control unit 120 of the designated area.

In the above example, the area on the image specified by the operator at the remote location is transmitted to the device control unit 120, which then converts it to distance, but it may also be converted to distance by the remote control unit 150 before transmitting it to the device control unit 120.

[Example of processing program]
The outline of the process has been made clear from the above explanation, but the specific process steps will be explained below using a flowchart.

[Information presentation management processing]
11 is a flowchart showing an example of the procedure for information presentation management processing as part of a program executed by the device control unit 120. This information presentation management processing can be executed, for example, as one thread of timer interrupt processing by the CPU of the device control unit 120. The example procedure will be described below.

Step S100: The device control unit 120 performs a communication execution process in the communication processing unit 134. This process is for performing mutual communication with the remote control unit 150 via the communication network 160.
Step S102: The device control unit 120 checks whether distance measurement is currently being performed. If distance measurement is not currently being performed (No), the device control unit 120 proceeds to step S104. If distance measurement is already being performed (Yes), the device control unit 120 skips step S104 and proceeds to step S106.

Step S104: If distance measurement is not yet in progress, the device control unit 120 determines whether the start condition is met. For example, the determination here can be triggered by the remote control unit 150 informing the user of the start of a carrying operation, and the start condition being considered to be met. If the start condition is not met (No), the device control unit 120 temporarily exits from this processing (Return). On the other hand, if the start condition is met (Yes), the device control unit 120 proceeds to step S106.

Step S106: The equipment control unit 120 executes inter-bucket distance measurement processing in the inter-bucket distance measurement unit 124. In this processing, bucket BK is detected and the distance is measured using several frames of images captured by the fisheye cameras 104, 106 as described above. The specific processing content will be described later using another flowchart.

Step S108: The equipment control unit 120 checks whether measurement of the inter-bucket distance Db has been completed. If the measurement process is still in progress (No), the equipment control unit 120 temporarily exits this process (Return). On the other hand, if it determines that measurement has been completed (Yes), step S110 is executed next.

Step S110: The device control unit 120 executes object distance measurement processing in the object distance measurement unit 128. In this processing, the distance to the specified object is measured using several frames of images captured by the fisheye cameras 104 and 106 as described above, or the measurement is performed using the distance sensor 102. The specific processing content will be described later using another flowchart.

Step S112: The device control unit 120 checks whether measurement of the object-to-object distance Do has been completed. If the measurement process is still in progress (No), the device control unit 120 temporarily exits this process (Return). On the other hand, if it determines that measurement has been completed (Yes), step S114 is executed next.

Step S114: The equipment control unit 120 executes the bucket-object distance measurement process using the bucket-object distance measurement unit 130. In this process, the distance Db-o is calculated as described above.

Step S116: The device control unit 120 executes a presentation information generation process in the presentation information processing unit 132. In this process, information to be presented is generated in a format corresponding to the calculated distance Db-o.

Step S118: The equipment control unit 120 determines whether the termination condition has been met. For example, the determination here can be triggered by the remote control unit 150 notifying the end of the transport operation. The end of the transport operation may be explicitly notified by the operator operating the operation unit 140, or the remote control unit 150 may automatically determine and notify the equipment control unit 120 of the end of the transport operation when the operator performs an operation other than the transport work, such as rotating the revolving unit TR or driving the running unit MV. If the termination condition is not met (No), the equipment control unit 120 exits this process (Return). On the other hand, if the termination condition is met (Yes), the equipment control unit 120 proceeds to step S120.

Step S120: The device control unit 120 executes information presentation termination processing to end presentation mode. In this processing, the device control unit 120 resets the memory of the various distances Db, Do, and Db-o that have been measured and calculated up to that point, and clears various condition flags. The device control unit 120 also switches from presentation mode to non-presentation mode in the program. The device control unit 120 then exits this processing (returns).

[Bucket distance measurement process]
12 is a flowchart showing an example of the procedure for the inter-bucket distance measurement process. This process is called during the information presentation management process (FIG. 11) and is executed as a subroutine in the inter-bucket distance measurement unit 124. The procedure will be described below along with the example procedure.

Step S200: When a call is first made, a process is performed to transition to the step that was being executed up until the previous time. In this process, the step number to be started in this call is determined based on the step number saved in the previous call. If this is the first call, the step number being executed has not yet been saved, so the bucket distance measurement unit 124 starts directly from step S202. On the other hand, if this is the second or subsequent call, the step number saved in the previous call is referenced to determine the step number to be started in this call.

Step S202: An image frame captured by the left front fisheye camera 104 is acquired. The acquired frame is stored in a temporary storage area, such as a frame buffer.

Step S204: The number of frames acquired so far is checked to determine whether the number of samples has reached a sufficient number for the difference calculation. If the number of acquired frames is not sufficient (No), the process proceeds to step S224.
Step S224: In this case, the step number currently being executed is saved, and the process temporarily exits the process and returns to the information presentation management process that called it (RETURN). Therefore, at the next call, it is determined in step S200 that the process will start from step S202.

If it is determined that the number of frames already acquired is sufficient in the next call or thereafter (step S204 = Yes), step S206 and subsequent steps are executed.
Step S206: Detect a moving object using the several frames already acquired. Here, as described above, an object corresponding to a moving object is extracted from the difference between several frames of images captured by the left front fish-eye camera 104.

Steps S208 to S212 are the same as those described above for the right front fish-eye camera 106, and the details thereof will be omitted.
Step S214: It is confirmed whether detection of moving objects using the left front and right front fisheye cameras 104, 106 has been completed. If detection of moving objects has not been completed (No), the step number is saved in step S224, and the process returns to the caller. In this case, the next time the process is called, it is determined in step S200 that the process will start again from step S202. On the other hand, if detection of moving objects has been completed (Yes), the process proceeds to step S216.

Step S216: Detect buckets BK from among the moving objects. Specifically, detect buckets BK from the image feature amounts of all the detected moving objects.
Step S218: Next, it is confirmed whether or not the detection of the bucket BK was successful. If a moving object is detected from several frames of images, but nothing corresponding to the bucket BK is detected from the image features (No), the step number is saved in step S224, and the process returns to the caller. In this case, at the next call, it is determined in step S200 that the process will start again from step S202. On the other hand, if the detection of the bucket BK was successful (Yes), the process proceeds to step S220.

Step S220: The left and right ends of the extracted bucket BK are detected from the image. Here, as shown in the example above (FIG. 8), the left and right ends of the horizontal edge EG are detected. Note that the left and right ends of the bucket BK may also be detected from edges other than the horizontal direction of the image, the overall contour, or other image feature quantities.
Step S222: Then, a distance Db to the bucket BK is calculated by stereo processing. The calculated distance Db is used in the subsequent information presentation management processing.
After the above steps are executed, the process returns to the information presentation management process that called the process (return).

[Object distance measurement process]
13 is a flowchart showing an example of the procedure for object distance measurement processing. This process is called during the information presentation management process (FIG. 11) and is executed as a subroutine in the object distance measurement unit 128. The example procedure will be described below.

Step S300: First, it is confirmed whether an object area has been specified from the remote control unit 150. If an object area has not yet been specified (No), the process exits and returns to the information presentation management process that called it. On the other hand, if an object area has been specified (Yes), step S302 is executed next.

Step S302: Set the object designated by the remote control unit 150. Here, as shown in the example above (FIG. 8), the image area designated by the touch operation is set as the object on the device control unit 120 side.
Step S304: Then, the distance Do to the set object is measured. As for the measurement method, as already mentioned, it may be either stereo processing using images captured by the fisheye cameras 104 and 106, or measurement using the distance sensor 102. The distance Do measured here is also used in the subsequent information presentation management process.
After the above steps are executed, the process returns to the information presentation management process that called the process (return).

[Remote operation management processing]
14 is a flowchart showing an example of the procedure for remote operation management processing as part of a program executed by the remote control unit 150. This remote operation management processing can be executed, for example, as one thread of timer interrupt processing by the CPU of the remote control unit 150. Furthermore, the remote operation management processing does not have a conditional branch within the main thread, and is composed of a group of multiple subroutines. The subroutines will be described below.

Step S400: The remote control unit 150 executes operation input processing as a subroutine. Here, processing is mainly based on the operator's operations on the backhoe BH. The specific content of the processing will be described later using a separate flowchart.

Step S500: The remote control unit 150 executes a communication execution process as a subroutine, in which various pieces of information to be transmitted to the device control unit 120 are transmitted.
Step S600: The remote control unit 150 also executes a display output process as a subroutine, in which an image signal to be displayed on the display unit 144 (touch screens TS1 and TS2) is output.
Step S700: The remote control unit 150 then executes a sound output process as a subroutine, in which sound or voice is output from the sound output unit 142 (a speaker, a headset, etc., not shown).

By the remote control unit 150 constantly executing the above subroutines, overhead images, forward images, sound output, etc. are displayed in the operation room R at any time, and remote operation of the backhoe BH is realized. Furthermore, when the work mode switches to transport mode, information is presented to the operator based on information from the equipment control unit 120 in a manner that conveys a sense of distance.

[Operation input processing]
15 is a flowchart showing an example of the procedure of the operation input process executed as a subroutine. This process is routinely called and executed in the remote operation management process (FIG. 14) as described above.

Step S402: The remote control unit 150 selects a work mode for operating the backhoe BH. Work modes can be broadly divided into, for example, a "transport work mode" and an "other work mode." The selection of the work mode is based on the operation performed by the operator on the operation unit 140 (controller CN). As described above, when the operator begins the operation to lower the bucket BK, the remote control unit 150 selects the "transport work mode." On the other hand, when the operator performs an operation to drive the traveling body MV or rotate the rotating body TR, the remote control unit 150 selects the "other work mode." Note that the "other work mode" may have further subdivided modes.

Step S404: The remote control unit 150 checks whether the "Transportation Work Mode" was selected in the previous step S402. If the "Transportation Work Mode" was actually selected (Yes), step S406 is then executed. On the other hand, if the "Other Work Mode" was selected (No), step S406 is skipped and the process proceeds to step S408.

Step S406: The remote control unit 150 executes an input to designate the area of the transported object. Here, as described above, the area of the object is designated in response to the operator's touch operation on the touch input unit 146 (touch screen TS2).

Step S408: The remote control unit 150 executes the operation input during the selected mode. Here, for example, if the currently selected mode is "other work mode," operation input is performed for the traveling unit MV and operation input is performed for the revolving unit TR. On the other hand, if the currently selected mode is "transport work mode," operation input is performed for the front working mechanism FR, including the bucket BK.

Step S410: The remote control unit 150 then generates transmission information for the device (device control unit 120). Here, an operation signal for the device control unit 120 is generated based on the various operation inputs performed in the previous step S408. Furthermore, if the "transportation work mode" is selected, information for transmitting "start of transportation work" is also generated.
After the above procedure is executed, the remote control unit 150 returns to the remote operation management process of the caller.

While the above is considered to be the first embodiment of system 100, the following second embodiment can also be applied to system 100.

Second Embodiment
16 is a diagram showing the arrangement of various devices in the second embodiment of the system 100. In the first embodiment, four fisheye cameras 104 to 110 are arranged at the four corners (front left and right corners, rear left and right corners) of the rotating unit TR, so that the bucket BK is imaged by two fisheye cameras 104 and 106.

In contrast, the arrangement in the second embodiment places four fisheye cameras 204-210 on the front, left and right sides, and rear of the rotating unit TR. Even in this case, the images captured by the four fisheye cameras 204-210 can be combined to display an overhead image in the operation room R. However, with this arrangement, the bucket BK is only imaged by the single fisheye camera 204 attached to the front.

Therefore, in the second embodiment, although it is no longer possible to measure the distance Db to the bucket BK by stereo processing, the distance Db can be measured by setting the actual horizontal length of the bucket BK in advance at the initial setting stage.
That is, compared to the first embodiment, the only difference in the second embodiment is that instead of performing stereo processing, the length of the bucket BK on the image and its actual length are used to obtain the result, and the rest can be performed in the same way as in the first embodiment.

The information presentation system 100 of the above embodiment can present information that allows the operator to easily grasp distances when performing tasks such as transporting an object, even at a remote location separate from the actual work site.

The present invention is not limited to the above-described embodiment, and can be implemented in various modified forms.
As already mentioned, the construction machine to which the system 100 is applied may be other than the backhoe BH, and the system 100 may also be applied to various types of work equipment other than construction machines, such as a self-propelled remote-controlled robot, a rescue robot, or a manipulator used outside the atmosphere. In these cases, the distance between the working body (robot hand, etc.) of the various types of work equipment and the target object can be measured, and information can be presented in a manner that makes it easy to convey the sense of distance at a remote location.

The arrangement, number, angle of view, orientation, etc. of the fisheye cameras 104-110 may be other than those exemplified in the first and second embodiments. Furthermore, if the distance to the target object is measured using stereo processing using the fisheye cameras 104 and 106, the distance sensor 102 does not need to be installed.

In the above embodiment, only one distance sensor 102 is provided, but multiple distance sensors 102 may also be provided.

In the above embodiment, the device control unit 120 generates the information to be presented, but the device control unit 120 may not generate the information, and instead transmit only the image signal and distance Db-o information to the remote control unit 150, which then generates the information to be presented in a format corresponding to the distance Db-o.

Alternatively, the device control unit 120 may not measure the distance Db-o, but may instead transmit all necessary image signals and detection results to the remote control unit 150, which may then measure the distances Db, Do, and Db-o. In this case, the functional blocks of the device control unit 120 would be relocated to the remote control unit 150. Alternatively, both the device control unit 120 and the remote control unit 150 may have equivalent functional blocks, and it may be decided on a case-by-case basis which side will measure the distances and generate information.

In addition, the structures illustrated in the embodiments are merely preferred examples, and it goes without saying that the present invention can be suitably implemented by adding various elements to the basic structure or by substituting parts.

REFERENCE SIGNS LIST 100 Information presentation system 120 Equipment control unit 124 Bucket distance measurement unit 126 Object setting unit 128 Object distance measurement unit 130 Bucket-object distance measurement unit 132 Presentation information generation unit 142 Sound output unit 144 Display unit 146 Touch input unit 150 Remote control unit

Claims (8)

an object setting means for setting an object to be specified as an object to be transported and loaded onto the movable working body;
a distance measuring means for measuring the distance between the working body and the object;
a presentation means for presenting information in a manner according to the distance measured by the distance measurement means ,
The object setting means
The work range of the work body includes a display unit that allows touch operation on a screen that displays an image of the work range, and the target object can be set by directly touching an area in the image on the screen of the display unit where the load to be transported is located.
An information presentation system comprising : The information presentation system according to claim 1,
The distance measuring means
measuring a distance between the working body and the object that changes dynamically during the process of the work performed by the working body;
The presentation means
An information presentation system characterized in that a manner in which information is presented is changed in accordance with a change in the distance dynamically measured by said distance measuring means. 3. The information presentation system according to claim 1,
The distance measuring means
an object distance measuring means for measuring a distance from a predetermined reference position to the object;
a work body distance measuring means for measuring a distance from the reference position to the work body,
an information presentation system, characterized in that the distance between the working body and the object is measured based on the distances measured by the inter-object distance measuring means and the inter-working body distance measuring means; 4. The information presentation system according to claim 1,
The distance measuring means
a stereo camera that captures an image of the workpiece from a predetermined reference position;
an information presentation system, characterized in that the distance from the reference position to the work object is measured based on an image of the work object captured by the stereo camera; 4. The information presentation system according to claim 1,
The distance measuring means
an actual length setting means for initially setting an actual horizontal length of a lower edge portion of the working body;
a camera that captures an image of the workpiece from a predetermined reference position;
An information presentation system characterized by calculating the distance from the reference position to the work body based on the horizontal length of the lower edge portion of the work body that appears in the image captured by the camera and the set actual length. 6. The information presentation system according to claim 1,
The distance measuring means
a distance sensor capable of measuring a distance from a predetermined reference position;
An information presentation system, characterized in that the distance between the working body and the object is measured using the distance to the object measured by the distance sensor. 7. The information presentation system according to claim 1,
The presentation means
The display device has a presentation mode in which information is presented and a non-presentation mode in which information is not presented,
An information presentation system characterized in that switching between the presentation mode and the non-presentation mode is triggered by an operation specifying the transportation of the object as work to be performed by the work body. The information presentation system according to claim 7 ,
The presentation means
The information presentation system is characterized in that switching to the non-presentation mode is triggered by an operation specifying an operation other than that performed by the working object during the presentation mode.