JP7569261B2 - Remote Control System - Google Patents

Description

The present invention relates to a remote control system.

In disaster recovery work following an earthquake, flood, landslide, etc., when there is a risk of secondary disaster due to work using a manned work machine, work may be performed using a remotely operated unmanned work machine. For example, Patent Literature 1 describes a remote control system that reproduces binocular parallax and displays images from two fisheye cameras installed on a work machine on a head mounted display (HMD) worn by an operator in a remote control room.
According to the remote control system described in Patent Document 1, binocular parallax provides a sense of depth and perspective that cannot be obtained with a monocular camera. In addition, a fisheye lens is used, providing a wide field of view, and since the direction of the head-mounted display and the direction of the image are synchronized, the user can view the image in the direction they want by shaking their head wearing the head-mounted display left and right.

JP 2018-121195 A

However, in conventional systems, it was not necessarily easy to check the rear when remotely controlling a work machine. Even with a camera using a fisheye lens, it can only capture images up, down, left, and right to 180 degrees in the front, and the rear of the work machine is outside the capture range of the camera using a fisheye lens, so it is not possible to capture the rear of the work machine. In addition, a common method of checking the rear is to grasp the situation using a rearview mirror (such as a rearview mirror or side mirror), but since the camera in the above system is installed in the driver's seat and the rearview mirror of the work machine is also within the capture range, it is possible to check the mirror in the image captured by the fisheye lens in the above system. Here, the rearview mirror is generally installed in a location slightly away from the front so as not to interfere with work. And the resolution of the image captured using a fisheye lens decreases toward the outer periphery. Therefore, the rear image of the rearview mirror through the image captured using the fisheye lens has low image quality and is insufficient as an image for rearview confirmation.
From this viewpoint, the present invention provides a remote control system that makes it easy to check the rear of a work machine during remote control.

The remote control system according to the present invention comprises a remote operation device for remotely operating a work machine, a forward stereo camera equipped with a fisheye lens and installed in the driver's cab of the work machine , a rear camera equipped with a fisheye lens for capturing images behind the work machine, and a head mounted display equipped with an angle sensor for displaying images captured by the forward stereo camera and the rear camera.
The fisheye lens of the front stereo camera is a circular fisheye lens, and the fisheye lens of the rear camera is a diagonal fisheye lens.
A partial image in a desired direction is extracted from a wide-range fisheye image captured by the front stereo camera in accordance with the direction of the operator's head detected by the angle sensor, and a corrected image in which distortion of the partial image is corrected is displayed on the head-mounted display by reproducing binocular parallax. A rear camera image captured by the rear camera is displayed superimposed on or included in the corrected image in a small window format .
"Overlapping the rear camera image on the corrected image" means displaying the rear camera image in an area where the corrected image data exists. "Including the rear camera image in the corrected image and displaying it" means displaying the rear camera image in an area where the corrected image data does not exist (such as a cutout area or margin) .
In the remote control system according to the present invention, the operator can easily check the situation behind the work machine via the rear camera image. Note that the rear camera image is displayed superimposed on or included in the corrected image obtained by correcting the distortion of the image captured by the front stereo camera, so that the distortion correction does not affect the image quality of the rear camera image, and therefore the image can have sufficient image quality for checking the rear.

The position of the rear camera image relative to the fisheye image may be fixed and displayed regardless of the direction of the operator's head. In this case, the rear camera is installed in place of a rearview mirror provided on the work machine or adjacent to the mirror, and the rearview camera image is displayed in a portion of the corrected image that corresponds to the rearview mirror.
In this way, the front area, which is important for performing work in the fisheye image, is not obstructed by the rear camera image. Also, since the operator can check the rear in the same way as if he/she were on board the work machine, the operator can operate the work machine in a situation that is closer to the situation when he/she is actually on board, even though the operation is remote. Another advantage is that it is easy to see which direction is in front of the work machine.
The position of the rear camera image relative to the fisheye image may be moved according to the direction of the operator's head, and the position of the rear camera image relative to the corrected image may be fixed and displayed. In this case, it is preferable to display the rear camera image near the edge of the corrected image.
In this way, the rear camera image is always displayed regardless of the direction of the operator's head (i.e., the rear camera image never leaves the operator's field of vision). In addition, since the position of the rear camera image can be moved, no part of the fisheye image is invisible.
In addition, it may be possible to select between displaying the rear camera image with the position of the rear camera image fixed relative to the fisheye image and displaying the rear camera image with the position of the rear camera image moved relative to the fisheye image.
This allows the operator to select the display method according to the work situation, making remote control easier.

The present invention makes it easy to check the rear of a work machine when remotely controlled.

1 is a schematic configuration diagram of a remote control system according to an embodiment of the present invention. 1A and 1B are diagrams for explaining the imaging range of a camera provided in a remote control system according to an embodiment of the present invention, where FIG. FIG. 11 is a diagram for explaining distortion correction of an image captured using a fisheye lens, where (a) is an example of a subject, (b) is an image of an image captured with a fisheye lens, (c) is an image of a distortion-corrected image, (d) shows the image sensor grid before distortion correction, and (e) shows the image sensor grid after distortion correction. 1A is a diagram illustrating a diagonal fisheye lens, and FIG. 1B is a diagram illustrating a circumferential fisheye lens. 5 is a diagram for explaining the wipe position fixing method. Fig. 5(a) is an image of an image seen by an operator facing forward, and (b) is a diagram for explaining the positional relationship of each image in that state. Fig. 5(c) is an image of an image seen by an operator facing upward, and (d) is a diagram for explaining the positional relationship of each image in that state. Fig. 5(e) is an image of an image seen by an operator facing left, and (f) is a diagram for explaining the positional relationship of each image in that state. 6 is a diagram for explaining the wipe position tracking method. Fig. 6(a) is an image of an image seen by an operator facing forward, and (b) is a diagram for explaining the positional relationship of each image in that state. Fig. 6(c) is an image of an image seen by an operator facing upward, and (d) is a diagram for explaining the positional relationship of each image in that state. Fig. 6(e) is an image of an image seen by an operator facing left, and (f) is a diagram for explaining the positional relationship of each image in that state. 1A to 1C are diagrams for explaining attitude control of a head mounted display. FIG. 13 is a diagram for explaining coordinate transformation by rotation in the yaw direction. FIG. 11 is a diagram for explaining the setting of position coordinates of stereo camera images in a wipe position tracking method, in which (a) is an example of setting the stereo camera images, (b) shows a state in which the camera faces the reference direction (a direction of "0 degrees" relative to the x-axis), and (c) shows a state in which the camera faces diagonally forward and to the left (a direction rotated "45 degrees" counterclockwise relative to the x-axis). FIG. 11 is a diagram for explaining the setting of position coordinates for the rear camera image (wipe) and the calculation of position coordinates after a rotation operation, in which (a) is an example of the setting of the wipe of the rear camera, (b) shows a state facing the reference direction (a direction of "0 degrees" relative to the x-axis), and (c) shows a state facing diagonally forward and to the left (a direction rotated "45 degrees" counterclockwise relative to the x-axis).

Below, the form for carrying out the present invention will be described in detail with reference to the drawings as appropriate. Each figure is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to the illustrated examples. In each figure, common or similar components are given the same reference numerals, and duplicate explanations thereof will be omitted.

<Configuration of remote control system according to embodiment>
The configuration of a remote control system 1 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic configuration diagram of the remote control system 1. The remote control system 1 is a system for remotely operating a work machine 2. The remote control system 1 can be used in various situations where the work machine 2 is used, and there are no particular limitations on the type of work machine 2 to be remotely operated or the work to be performed using the work machine 2.
As shown in Fig. 1, the remote control system 1 is mainly composed of devices 10 mounted on a work machine 2 and devices 20 provided in an operation room. The work machine 2 may be, for example, a machine that can be operated by an operator while riding there under normal circumstances (one that has a driver's cab, etc.) that has been modified to be remotely operable. The work machine 2 is, for example, a hydraulic excavator (backhoe), a vibrating roller, a bulldozer, a dump truck, a crane, etc., which are types of construction machinery.
The devices 10 on the work machine side include a forward stereo camera 11 and at least one (two in FIG. 1) rear camera 12. The devices 20 on the control room side include a remote control device 21, a head mounted display 22, and a processing device 23. Note that "front/back,""up/down," and "left/right" in the explanation of the devices 10 on the work machine side follow the arrows in FIG. 1. These directions correspond to the worker on board the work machine 2 (i.e., the operator who remotely controls it).
The remote control system 1 extracts a partial image of a region desired by the operator from a wide range of fisheye images captured by the forward stereo camera 11 according to the direction and movement of the operator's head detected by an angle sensor of the head mounted display 22, and projects a corrected image in which the distortion of the partial image is corrected on the head mounted display 22. The remote control system 1 is also characterized in that the rear camera image captured by the rear camera 12 is displayed superimposed or included in the corrected image in a small window format (wipe). Displaying the rear camera image superimposed on the corrected image means displaying the rear camera image in an area where the data of the corrected image exists. Displaying the rear camera image included in the corrected image means displaying the rear camera image in an area where the data of the corrected image does not exist (a cut-out part or a margin). In terms of clarifying the relationship between the corrected image and the rear camera image, it is desirable to place the rear camera image within the area of the corrected image.

(Equipment on the work machine side)
The devices 10 on the work machine side will be described with reference to Fig. 1. The front stereo camera 11 is equipped with two cameras 11a arranged on the left and right. The front stereo camera 11 is installed, for example, in the driver's cab of the work machine 2, and captures images used for remote operation of the work machine 2. The two cameras 11a, 11a equipped on the front stereo camera 11 correspond to the left and right eyes of an operator who is on board under normal circumstances. The camera 11a has a resolution that allows the situation at the work site to be recognized, and preferably has a frame rate that does not interfere with remote operation. The camera 11a is preferably a video camera with a resolution of 4K or more, for example.
Each camera 11a constituting the forward stereo camera 11 is equipped with a fisheye lens. The fisheye lens is a lens that enables shooting in a super-wide angle range (for example, 180°). The imaging range K1 of the forward stereo camera 11 is shown in FIG. 2. FIG. 2 is a diagram for explaining the imaging range of the camera equipped in the remote control system 1, (a) is a plan view, and (b) is a side view. The forward stereo camera 11 always captures the super-wide angle imaging range K1 with fisheye images. Although the fisheye image can display a wide range of areas, the distortion becomes larger toward the periphery. In this embodiment, a partial image corresponding to the direction of the operator's head is extracted from the fisheye image that is always captured, and a distortion-free corrected image is created from this partial image. The corrected image can also be created by other methods. The image processing for creating the corrected image is preferably performed by a processing device 23 installed in the operation room. By performing image processing using a processing device 23 installed in the operator's room, it is possible to speed up image processing, simplify the configuration of the device to be installed on the work machine 2, and reduce the weight of the head-mounted display 22.

The distortion correction of an image captured using a fisheye lens will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining the distortion correction of an image captured using a fisheye lens, in which (a) is an example of a subject, (b) is an image of an image captured by the fisheye lens, (c) is an image of a distortion-corrected image, (d) shows the image sensor grid before distortion correction, and (e) shows the image sensor grid after distortion correction.
For ease of explanation, a checkerboard pattern (a checkerboard pattern of different colored checkerboard patterns) is assumed as the subject (see FIG. 3(a)). When the subject shown in FIG. 3(a) is photographed with a camera equipped with a fisheye lens (fisheye camera), straight lines are curved in the acquired image as shown in FIG. 3(b). Therefore, it is necessary to correct the distortion when displaying the image on the head-mounted display, and by performing the distortion correction, the subject returns to the state before it was distorted by the fisheye lens as shown in FIG. 3(c).
On the other hand, as shown in Fig. 3(d), the image sensor of the camera is in a lattice pattern, and when a distorted subject in an image captured by a fisheye camera is corrected, the correction is performed for each dot of the image sensor. In Fig. 3(d), the dots in the center are labeled with the symbol D1, and the dots at the ends are labeled with the symbol D2. As shown in Fig. 3(e), in the distortion-corrected image, the amount of distortion of the dots of the image sensor is small in the center and large at the ends. In other words, in the distortion-corrected image, the dot density is large in the center and small at the ends. Therefore, when viewed on a head-mounted display, the resolution is lower toward the ends compared to the center of the displayed image.

The rear camera 12 captures images used for rear confirmation of the work machine 2. The rear camera 12 may be installed in a position where it can capture images necessary for rear confirmation, and may be installed, for example, in place of a rear confirmation mirror or adjacent to the mirror. As shown in FIG. 2(a), in this embodiment, the rear camera 12 is installed facing rearward at the position of the side mirror. The rear camera 12 is fixed to the left and right frames of the driver's seat. As shown in FIG. 2(a), the right rear camera 12R mainly captures an image range K2 of the right rear of the work machine 2. On the other hand, the left rear camera 12L mainly captures an image range K2 of the left rear of the work machine 2. Rear confirmation is mainly performed when the work machine 2 is reversing or turning, so the rear camera 12 is preferably attached at an angle that allows it to capture an image of the tracks of the work machine 2 (see FIG. 2(b)).
The rear camera 12 may be equipped with a fisheye lens. It is known that there are two types of fisheye lenses: a diagonal fisheye lens and a circular fisheye lens. The differences between the types of fisheye lenses will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining the types of fisheye lenses, where Fig. 4(a) is a diagram for explaining a diagonal fisheye lens and Fig. 4(b) is a diagram for explaining a circular fisheye lens.
As shown in Fig. 4(a), in a diagonal fisheye lens, the diameter of the image circle is the diagonal line of the camera sensor. With a diagonal fisheye lens, the subject appears large (more pixels = better image quality), but many areas are cut off, so it is not suitable for viewing a wide range. On the other hand, as shown in Fig. 4(b), in a circular fisheye lens, the subject appears small (fewer pixels = lower image quality), but no areas are cut off, so it is suitable for viewing a wide range.
When operating a work machine 2 (particularly when operating a construction machine), it is desirable to have a wide viewing angle in all directions, so it is desirable to use a circular fisheye lens in the forward stereo camera 11 of the remote control system 1. On the other hand, a viewing angle that is as wide is not required for checking the rear, so it is desirable to use a diagonal fisheye lens, which has better image quality.

(Equipment in the control room)
The devices 20 in the control room will be described with reference to Fig. 1. The remote control device 21 is a device for remotely controlling the work machine 2. In Fig. 1, a joystick type controller is shown as an example of the remote control device 21, but the remote control device 21 is not limited to this. The remote control device 21 may include a foot pedal or the like. The remote control device 21 transmits operation information for the work machine 2 input by the operator to the work machine 2 via a processing device 23, for example.
The head mounted display 22 displays the images captured by the front stereo camera 11 (sometimes referred to as "stereo camera images" or "fisheye images") and the images captured by the rear camera 12 (sometimes referred to as "rear camera images"). The images captured by the front stereo camera 11 and the rear camera 12 are transmitted to the head mounted display 22 via, for example, a processing device 23.
The head mounted display 22 is worn by an operator in an operating room. Images from the two cameras 11a, 11a constituting the forward stereo camera 11 are projected on the left eye and right eye of the head mounted display 22 while reproducing binocular parallax. That is, the forward stereo camera 11 captures images of an object from different angles using two fisheye lenses, and the operator can see an image (3D image) with a sense of depth that reproduces binocular parallax. That is, the remote control system 1 can faithfully project the field of view from the cab of the work machine 2 onto the head mounted display 22 in almost real time. The operator can work while viewing the image with a sense of depth in almost real time, and therefore can accurately operate the work machine 2.
Further, the head mounted display 22 displays an image from the rear camera 12 (rear camera image). The rear camera image is displayed superimposed on the stereo camera image or included as part of the stereo camera image. Displaying the rear camera image superimposed on the stereo camera image means displaying the rear camera image in an area where the data of the stereo camera image exists. Displaying the rear camera image included in the stereo camera image means displaying the rear camera image in an area where the data of the stereo camera image does not exist (a cut-out part or a margin). In terms of clarifying the relationship between the stereo camera image and the rear camera image, it is desirable to arrange the rear camera image within the area of the stereo camera image. A method of displaying the rear camera image will be described later.

The head mounted display 22 includes an angle sensor, a gyro sensor, etc. (not shown), and detects the direction, movement, etc. of the operator's head by the angle sensor, the gyro sensor, etc. The detected direction, movement, etc. of the operator's head are used to link with the image projected on the head mounted display 22. The type and number of sensors included in the head mounted display 22 are not particularly limited as long as they can link the operator wearing the head mounted display 22 with the image to be projected.
The processing device 23 is a computer that performs overall control of the remote control system 1. The processing device 23 has, for example, a function related to image display for displaying an image captured on the work machine 2 side on the head mounted display 22, and a function related to remote operation for controlling the work machine 2 by operation information input via the remote operation device 21. These functions of the processing device 23 are realized, for example, by a central processing unit (CPU) reading and executing a predetermined program from a read only memory (ROM) (not shown). The processing device 23 is connected to the remote operation device 21 and the head mounted display 22, for example, via a universal serial bus (USB) or a display port, and communicates with the devices 10 on the work machine side via wireless communication. The location of the processing device 23 is not limited to the operation room, and it can be arranged in a place other than the operation room. It is also possible to configure the processing device 23 so that some of the functions of the processing device 23 are possessed by a computer provided in the work machine 2.

<How each image is displayed using a head-mounted display>
The head mounted display 22 displays the rear camera image in a small window format (wipe). The rear camera image is displayed in a portion that does not interfere with the stereo camera image. The rear camera image may be fixed in position relative to the stereo camera image (for example, displayed at the top when facing forward), or may not be fixed in position relative to the stereo camera image (for example, moved to follow the movement of the head wearing the head mounted display 22, and displayed in the same position in the operator's field of view regardless of the direction the operator faces). The former method is called the "fixed wipe position method" and the latter method is called the "following wipe position method". It is preferable that the display method of the rear camera image be selectable according to the operator's preference.

The image display methods ("fixed wipe position method" and "following wipe position method") using a head mounted display will be described with reference to Figs. 5 and 6.
(fixed wipe position method)
Fig. 5 is a diagram for explaining the wipe position fixing method. Fig. 5(a) is an image of an image seen by an operator facing forward, and (b) is a diagram for explaining the positional relationship of each image in that state. Fig. 5(c) is an image of an image seen by an operator facing upward, and (d) is a diagram for explaining the positional relationship of each image in that state. Fig. 5(e) is an image of an image seen by an operator facing left, and (f) is a diagram for explaining the positional relationship of each image in that state.
Reference symbol E in FIG. 5 indicates a stereo camera image (fisheye image), reference symbol WR indicates a wipe showing the image from the right rear camera, and reference symbol WL indicates a wipe showing the image from the left rear camera.
In the fixed wipe position method, the positions of the wipes WR and WL of the rear camera 12 are fixed with respect to the stereo camera image E. Therefore, as shown in Figures 5(b), (d), and (f), even if the operator changes the direction in which he or she faces, the wipes WR and WL of the rear camera 12 do not move with respect to the stereo camera image E. Therefore, the wipes WR and WL do not obstruct the important front portion of the stereo camera image E. However, depending on the direction in which the operator faces, the wipes WR and WL of the rear camera 12 may be out of sight (see Figure 5(f)).

(Wipe position tracking method)
Fig. 6 is a diagram for explaining the wipe position tracking method. Fig. 6(a) is an image of an image seen by an operator facing forward, and Fig. 6(b) is a diagram for explaining the positional relationship of each image in that state. Fig. 6(c) is an image of an image seen by an operator facing upward, and Fig. 6(d) is a diagram for explaining the positional relationship of each image in that state. Fig. 6(e) is an image of an image seen by an operator facing left, and Fig. 6(f) is a diagram for explaining the positional relationship of each image in that state.
Reference symbol E in FIG. 6 indicates a stereo camera image (fisheye image), reference symbol WR indicates a wipe showing the image from the right rear camera, and reference symbol WL indicates a wipe showing the image from the left rear camera.
In the wipe position tracking method, the positions of the wipes WR and WL of the rear camera 12 are not fixed with respect to the stereo camera image E. Therefore, as shown in Fig. 6 (b), (d), and (f), when the operator changes the direction of his/her head, the wipes WR and WL of the rear camera 12 move to follow the head movement. Therefore, the wipes WR and WL of the rear camera 12 do not go out of the field of view regardless of the direction of the operator's head. In addition, since the position of the rear camera image can be moved, no part of the stereo camera image E is invisible.

<How to set up rear camera footage>
A method of installing the rear camera image in the fixed wipe position method and the follow wipe position method will be described with reference to FIG. 7 to FIG.
The head mounted display 22 is equipped with an acceleration sensor, a geomagnetic sensor, etc., and is therefore capable of detecting three-axis rotation (roll, pitch, yaw) and recognizing its own attitude information (3DoF control) as shown in Fig. 7. Fig. 7 is a diagram for explaining the attitude control of the head mounted display 22.
(fixed wipe position method)
In the fixed wipe position method, the coordinates of both the stereo camera image (fisheye image) and the rear camera image are given constants and are fixed, so that the image positions do not change even if the head mounted display 22 (the operator's head) is moved.

(Wipe position tracking method)
In the wipe position tracking method, the coordinates of the stereo camera images (fisheye images) are given constants to make them fixed coordinates, but the coordinates of the rear camera images are given variables including the angle information of the head mounted display 22. In other words, the rear camera images are not set to fixed coordinates. For example, by incorporating the attitude information of the head mounted display 22 as a variable into the position coordinates of the rear camera images, the rear camera images are always displayed even if the viewing direction is changed by moving the head mounted display 22.
For simplicity, the control of the wipe position tracking method in the yaw direction rotation will be described using two-dimensional coordinates (see FIG. 8). FIG. 8 is a diagram for explaining coordinate transformation by yaw direction rotation. On the two-dimensional coordinate plane shown in FIG. 8, the coordinates (X, Y) of a new point P obtained by rotating a point P(x, y) counterclockwise by θ around the origin O can be expressed by the following equation.
・X=xcosθ−ysinθ
・Y=xsinθ+ycosθ

Setting of position coordinates for stereo camera image E (fisheye image) will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining setting of position coordinates for stereo camera image E, in which (a) is a setting example of stereo camera image E, (b) shows a state facing the reference direction (direction of "0 degrees" with respect to the x-axis), and (c) shows a state facing diagonally forward left (direction rotated "45 degrees" counterclockwise with respect to the x-axis).
As shown in Fig. 9(a), in a coordinate system with the viewpoint position as the origin, a hemisphere of radius R centered on the origin is set, and the stereo camera image E is placed inside this hemisphere. When the operator faces the reference direction (the direction of "0 degrees" with respect to the x-axis), the visible range is, for example, as shown in Fig. 9(b). Even if the operator tries to look at the left side of the stereo camera image E and changes the viewing direction by rotating the head mounted display 22 counterclockwise as shown in Fig. 9(c), the stereo camera image E does not move and remains in the set state.

Next, the setting of position coordinates for the rear camera image (wipe W) and the calculation of position coordinates after rotation will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining the setting of position coordinates for the rear camera image (wipe W) and the calculation of position coordinates after rotation, in which (a) is an example of setting the wipe W of the rear camera 12, (b) shows a state facing the reference direction (a direction of "0 degrees" relative to the x-axis), and (c) shows a state facing diagonally forward left (a direction rotated "45 degrees" counterclockwise relative to the x-axis).
As shown in Fig. 10(a), in a coordinate system with the viewpoint position as the origin, the position coordinate P of the wipe W of the rear camera image is set as an "HMD rotation angle function". The position coordinate setting of the stereo camera image E is as described in Fig. 9. The rotation angle of the head mounted display 22 around the z axis is set as "θ", and the wipe position coordinate is set as P(xcosθ-ysinθ, xsinθ+ycosθ) (see the coordinate return formula described above). Then, the position coordinate P of the wipe W when the operator faces the reference direction (the direction of "0 degrees" with respect to the x axis) is as follows:
Position coordinates P of wipe W (xcos0°-ysin0°, xsin0°+ycos0°)=(x, y)
Moreover, when the operator faces diagonally forward and to the left (a direction rotated "45 degrees" counterclockwise about the x-axis), the position coordinate P of the wipe W is as follows:
Position coordinates P of wipe W (xcos45°-ysin45°, xsin45°+ycos45°)=((x-y)/√2, (x+y)/√2)
In this way, when the head mounted display 22 is rotated counterclockwise to change the viewing direction, the position of the wipe W is calculated by the "HMD rotation angle function" and the wipe W of the rear camera image is moved in accordance with the change in the viewing direction.

As described above, according to the remote control system 1 of this embodiment, the operator can easily check the situation behind the work machine 2 via the rear camera image. Note that the rear camera image is displayed superimposed on or included in the corrected image after the distortion of the image captured by the front stereo camera 11 has been corrected, so that the distortion correction does not affect the image quality of the rear camera image. In other words, the image quality of the rear camera image is not reduced by the distortion correction, and it is possible to provide an image of sufficient image quality for rear confirmation.
Furthermore, with the fixed wipe position method of this embodiment, the front portion of the stereo camera image E (fisheye image), which is important for performing work, is not obstructed by the rear camera image (wipe W). Furthermore, since the operator can check the rear in the same way as if he were on board the work machine 2, the operator can operate the work machine 2 in a situation that is close to the state when he is actually on board, even though the operation is remote. Another advantage is that it is easy to see which direction is in front of the work machine 2.
In addition, in the wipe position tracking method of this embodiment, the rear camera image (wipe W) is always displayed regardless of the direction of the operator's head (i.e., the rear camera image does not move out of the operator's field of vision). In addition, since the position of the rear camera image can be moved, no part of the stereo camera image E (fisheye image) is invisible.
It is also possible to select between a fixed wipe position method and a variable wipe position method, which allows the operator to select the display method according to the work situation, making remote control easier.

Although the embodiment of the present invention has been described above, the present invention is not limited to this, and can be practiced within the scope of the claims.
In this embodiment, the rear camera 12 is installed facing backward at the side mirror position, and the rear camera image is displayed at the side mirror position of the stereo camera image (fisheye image). However, the installation position of the rear camera 12 and the display location of the rear camera image are not limited to those described in the embodiment. For example, the rear camera 12 may be installed facing backward at the rear of the work machine 2, and the rear camera image may be displayed at the rear mirror position of the stereo camera image (fisheye image).

REFERENCE SIGNS LIST 1 Remote control system 2 Work machine 11 Front stereo camera 12, 12L, 12R Rear camera 21 Remote control device 22 Head mounted display 23 Processing device K1, K2 Imaging range E Stereo camera image (fisheye image)
W, WL, WR Wipe

Claims (6)

A remote control device for remotely controlling a work machine;
A forward stereo camera equipped with a fisheye lens and installed in a cab of the work machine;
a rear camera equipped with a fisheye lens and configured to capture an image of the rear of the work machine;
a head mounted display that is equipped with an angle sensor and displays images captured by the front stereo camera and the rear camera;
the fisheye lens of the forward stereo camera is a circular fisheye lens,
The fisheye lens of the rear camera is a diagonal fisheye lens,
A partial image in a desired direction is extracted from the wide-range fisheye image captured by the forward stereo camera in accordance with the direction of the operator's head detected by the angle sensor, and a corrected image obtained by correcting distortion of the partial image is displayed on the head-mounted display by reproducing binocular parallax;
A remote control system characterized in that a rear camera image captured by the rear camera is displayed in a small window format so as to be superimposed on or included in the corrected image. 2. The remote control system according to claim 1 , wherein the rear camera image is displayed at a fixed position relative to the fisheye image, regardless of the direction of the operator's head. The rear camera is installed in place of or adjacent to a rearview mirror equipped on the work machine,
3. The remote control system according to claim 2 , wherein the rear camera image is displayed in a portion of the corrected image that corresponds to the rearview mirror. a position of the rear camera image relative to the fisheye image is moved in accordance with a direction of the head of the operator, and the position of the rear camera image relative to the corrected image is fixed and displayed.
2. The remote control system according to claim 1 . 5. The remote control system of claim 4 , wherein the rear camera image is displayed near an edge of the corrected image. A selection can be made between displaying the rear camera image at a fixed position relative to the fisheye image and displaying the rear camera image at a movable position relative to the fisheye image.
2. The remote control system according to claim 1 .

Images