JP7792883B2 - Robot remote operation control device, robot remote operation control method, and program - Google Patents

Description

The present invention relates to a robot remote control control device, a robot remote control control method, and a program.

Generally, robots are equipped with a camera mounted on their heads that captures images of what is in front of them (see, for example, Patent Document 1). When remotely controlling such robots, a fisheye camera attached to the robot's head is used to present the operator with an image close to the operator's viewpoint (see, for example, Patent Document 2).

Japanese Patent Application Laid-Open No. 2022-46350 Patent No. 6940879

However, with the conventional technology described in Patent Document 2 and elsewhere, when an image mounted on the robot's head is used to control the robot while it is in operation, the robot's body can sometimes block the operator's view, making it impossible to operate an object properly.

The present invention was made in consideration of the above-mentioned problems, and aims to provide a robot remote operation control device, a robot remote operation control method, and a program that can reduce the operator's sense of discomfort when operating the robot.

(1) In order to achieve the above object, one aspect of the present invention provides a robot remote operation control device in which an operator remotely controls a robot, the device comprising: a spatial information sensor that acquires spatial information of the space in which the robot exists; an image output unit that presents the operator with a spatial reproduction image that reproduces the space in which the robot exists from the spatial information; a robot posture calculation unit that calculates the posture of the robot based on a three-dimensional model of the robot and the spatial information; and a spatial reproduction image rendering unit that calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, and renders an image in which spatial occlusion by the robot's body is removed as a spatial reproduction image.

(2) In order to achieve the above object, one aspect of the present invention provides a robot remote operation control device in which an operator remotely controls a robot, the robot remote operation control device comprising: a spatial information sensor that acquires spatial information of the space in which the robot exists; an image output unit that presents the operator with a spatial reproduction image that reproduces the space in which the robot exists from the spatial information; a robot posture calculation unit that calculates the posture of the robot based on a three-dimensional model of the robot and the spatial information, and calculates how the robot appears from the operator's viewpoint; and a spatial reproduction image rendering unit that calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, and renders an image from any viewpoint of the operator as a spatial reproduction image.

(3) Furthermore, a robot remote operation control device according to one aspect of the present invention is the robot remote operation control device described in (1) above, in which the spatial reproduction image rendering unit calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, determines whether the robot's body is blocking the operator's view, and, if the robot's body is blocking the operator's view, renders an image from any viewpoint of the operator as a spatial reproduction image.

(4) Furthermore, a robot remote operation control device according to one aspect of the present invention is a robot remote operation control device described in any one of (1) to (3) above, further comprising an information acquisition unit that acquires operator status information on the status of an operator operating the robot, and the robot posture calculation unit calculates the posture of the robot based on a three-dimensional model of the robot, the spatial information sensor, and the operator status information.

(5) Furthermore, a robot remote operation control device according to one aspect of the present invention is a robot remote operation control device described in any one of (1) to (4) above, in which the spatial information sensor has a known position, acquires data in a time series, and assigns a timestamp.

(6) Furthermore, a robot remote operation control device according to one aspect of the present invention is a robot remote operation control device described in any one of (1) to (5) above, in which the spatial reproduction image rendering unit calculates how the robot appears from the operator's viewpoint using a model learned using the spatial information and images from the operator's viewpoint, which are training data.

(7) Furthermore, a robot remote operation control device according to one aspect of the present invention is the robot remote operation control device described in (4) above, further comprising an operation amount determination unit that determines the operation amount of the robot based on the operator state information.

(8) To achieve the above-mentioned object, one aspect of the present invention provides a robot remote operation control method in which a robot remote operation control device, in which an operator remotely controls a robot, acquires spatial information of a space in which the robot exists, calculates the posture of the robot based on a three-dimensional model of the robot and the spatial information, calculates how the robot appears from the operator's viewpoint based on the spatial information and the calculated posture of the robot, and renders an image in which spatial occlusion by the robot's body has been eliminated as a spatial reproduction image and presents it to the operator.

(9) In order to achieve the above object, one aspect of the present invention provides a robot remote operation control method in which a robot remote operation control device, in which an operator remotely controls a robot, acquires spatial information of a space in which the robot exists, calculates the posture of the robot based on a three-dimensional model of the robot and the spatial information, calculates how the robot will appear from the operator's viewpoint, calculates how the robot will appear from the operator's viewpoint based on the spatial information and the calculated posture of the robot, and renders an image from any viewpoint of the operator as a spatial reproduction image and presents it to the operator.

(10) To achieve the above object, one aspect of the present invention provides a program that causes a computer of a robot remote control control device, which is used by an operator to remotely control a robot, to acquire spatial information about the space in which the robot exists, calculate the posture of the robot based on a three-dimensional model of the robot and the spatial information, calculate how the robot appears from the operator's viewpoint based on the spatial information and the calculated posture of the robot, and render an image in which spatial occlusion by the robot's body has been eliminated as a spatial reproduction image and present it to the operator.

(11) In order to achieve the above object, one aspect of the present invention provides a program that causes a computer of a robot remote operation control device, in which an operator remotely controls a robot, to acquire spatial information of the space in which the robot exists, calculate the posture of the robot based on a three-dimensional model of the robot and the spatial information, calculate how the robot appears from the operator's viewpoint, calculate how the robot appears from the operator's viewpoint based on the spatial information and the calculated posture of the robot, and render an image from any viewpoint of the operator as a spatial reproduction image and present it to the operator.

According to (1) to (11) above, by eliminating the obstruction caused by the robot's body, the operator's sense of discomfort when operating the robot can be reduced.

1 is a diagram illustrating an example of the configuration of a robot remote operation control system according to an embodiment. 1A and 1B are diagrams illustrating an example of a robot according to an embodiment and an example of a location where a camera is installed. FIG. 10 is a diagram illustrating an example of a data format of joint angle information according to the embodiment. FIG. 2 is a diagram illustrating an example of a data format of image information according to the embodiment. 10A to 10C are diagrams for explaining an example of processing performed by a spatial reproduction image rendering unit according to the embodiment. 10A to 10C are diagrams illustrating an example of input and output of a model included in a spatial reproduction image rendering unit according to the embodiment. 1 is a flowchart of a processing procedure performed by a robot remote operation control system according to an embodiment. FIG. 10 is a diagram showing an example of an image in which the arm of the robot blocks the line of sight of the operator. FIG. 10 is an image diagram showing an example of an image in which the robot's arm blocks the operator's line of sight. FIG. 10 is an image diagram of an example of an image synthesized so as to erase the occluded portion. FIG. 10 is a diagram showing an example of an image of how the robot appears when its body is removed.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized.
In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted.
Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[overview]
In this embodiment, when a robot is remotely operated, even if the robot's body (e.g., a hand or end effector) blocks the target object being worked on, an image that appears unblocked is provided to the operator. The image provided is synthesized using known information such as the robot's posture, three-dimensional model information of the robot, and information on the installation position of the imaging device.
According to this embodiment, by eliminating the obstruction caused by the robot's body, it is possible to reduce the sense of discomfort felt by the operator when operating the robot.

[Configuration of robot remote operation control system]
An example of the configuration of a robot remote operation control system will be described.
1 is a diagram showing an example of the configuration of a robot remote operation control system according to this embodiment. As shown in Fig. 1, the robot remote operation control system 9 includes, for example, a robot 1, a second image capturing unit 2 (spatial information sensor), an HMD 3, an operation detection unit 4, a robot remote operation control device 5, and a robot model storage unit 6.

The robot 1 includes, for example, a first image capture unit 11 (spatial information sensor), a sensor 12 (spatial information sensor), an arm 13, an end effector 14, and a drive unit 15.

The robot remote operation control device 5 includes, for example, an acquisition unit 51 , an operation amount determination unit 52 , a robot posture calculation unit 53 , a spatial reproduction image drawing unit 54 , a control unit 55 , a memory unit 56 , and an image output unit 57 .
The acquisition unit 51 includes, for example, an image acquisition unit 511, a sensor information acquisition unit 512, and an operator information acquisition unit 513 (information acquisition unit).

Robot 1 is, for example, a single-arm robot, a double-arm robot, a multi-arm robot, an arm robot, etc.

The first image capture unit 11 is, for example, a CCD (Charge Coupled Device) image capture device or a CMOS (Complementary Metal-Oxide-Semiconductor) image capture device. The first image capture unit 11 is equipped with, for example, a fisheye lens or a wide-angle lens. The first image capture unit 11 is attached, for example, to the head or end effector 14 of the robot 1. The first image capture unit 11 is equipped with, for example, an RGB camera that captures RGB (red-green-blue) images and a depth camera that captures depth images. The images captured by the first image capture unit 11 are spatial information of the space in which the robot exists. The position of the first image capture unit 11 attached to the robot 1 is known and is stored, for example, in the robot model storage unit 6. The first image capture unit 11 acquires data in chronological order and assigns a timestamp.

The sensor 12 is, for example, a six-axis sensor. The sensor 12 also detects the joint angles of each joint of the arm 13 and each joint of the end effector 14. The sensor 12 is a joint torque sensor attached to the joints of the robot 1. In this way, the sensor 12 detects sensor information used by the robot 1. Note that the first image capture unit 11 may also be one of the sensors 12. The sensor 12 acquires data in chronological order and assigns a timestamp.

An end effector 14 is connected to the tip of the arm 13.

The end effector 14 has at least two fingers. The end effector 14 may have three or more fingers.

The second image capture unit 2 is installed, for example, in the working environment of the robot 1. The second image capture unit 2 is equipped, for example, with an RGB camera that captures RGB (red-green-blue) images and a depth camera that captures depth images. The images captured by the second image capture unit 2 are spatial information of the space in which the robot exists. The installation position of the second image capture unit 2 is known and is stored, for example, in the robot model storage unit 6. The second image capture unit 2 acquires data in chronological order and assigns a timestamp.

The HMD 3 is, for example, a head-mounted display worn by the operator on the head. The HMD 3 includes, for example, an image display unit 31 that presents images to the operator, a gaze detection unit 32 that detects the operator's gaze, and a communication unit (not shown) that transmits and receives information to and from the robot remote operation control device 5.

The operation detection unit 4 detects operation instructions from the operator. The operation detection unit 4 is, for example, a data glove.

The robot model storage unit 6 stores, for example, a three-dimensional shape model of the robot 1 (including dimensions, etc.).

The robot remote operation control device 5 uses spatial information about the space in which the robot exists, operator status information indicating the status of the operator operating the robot 1, and a robot model to render an image in which spatial occlusion caused by the body of the robot 1 has been eliminated as a spatial reproduction image.

The image acquisition unit 511 acquires images captured by the first image capture unit 11 and the second image capture unit 2.

The sensor information acquisition unit 512 acquires the joint angles of each joint of the end effector 14 detected by the sensor 12 from the sensor 12. The sensor information acquisition unit 512 may also acquire images captured by the image capture unit (first image capture unit 11, second image capture unit 2) as sensor information. Alternatively, the sensor information acquisition unit 512 may acquire operator state information detected by the operation detection unit 4 as sensor information.

The operator information acquisition unit 513 acquires operator status information about the status of the operator operating the robot 1, for example, from the operation detection unit 4. The operator status information indicates how the operator wants the robot 1 to operate, or how the operator wants the robot 1 to operate. The operator information acquisition unit 513 may also acquire gaze information detected by the gaze detection unit 32.

The operation amount determination unit 52 determines the operation amount of the robot 1 based on the operator state information.

The robot posture calculation unit 53 calculates the posture of the robot using a three-dimensional model of the robot 1 and information acquired by the sensor information acquisition unit 512. Alternatively, the robot posture calculation unit 53 calculates the posture of the robot using a three-dimensional model of the robot 1, information acquired by the sensor information acquisition unit 512, and information acquired by the operator information acquisition unit 513.

The spatial reproduction image drawing unit 54 calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit 53, and determines whether the body of robot 1 is blocking the operator's view. If the body of robot 1 is blocking the operator's view, the spatial reproduction image drawing unit 54 draws an image from which the spatial occlusion caused by the body of robot 1 has been removed as a spatial reproduction image based on the spatial information and the information calculated by the robot posture calculation unit 53. The spatial reproduction image drawing unit 54 provides the generated image to the HMD 3.

The control unit 55 controls the operation of the robot 1 based on the operation amount determined by the operation amount determination unit 52.

The memory unit 56 stores, for example, values, programs, etc. necessary for controlling the robot 1. The memory unit 56 temporarily stores acquired information.

The image output unit 57 outputs a spatial reproduction image, which recreates the space in which the robot exists from the spatial information, to the HMD 3 so that the operator can see it.

[Robot configuration example]
First, an example of a robot and an example of a camera installation location will be described.
FIG. 2 is a diagram showing an example of a robot according to this embodiment and an example of the installation location of a camera.
The robot 1 includes, for example, a body 16, a head 17, arms 13 (13L, 13R), and hands (end effectors) 14 (14L, 14R). In the following description, the end effectors 14 are also referred to as hands 14.
The hand 14 has first image capturing units 11a and 11b attached to the wrist on the back side of the palm, for example. The head 17 has first image capturing units 11c and 11d.
In addition, a second imaging unit 2 is installed in the working space of the robot 1 .

The first image capturing units 11 c and 11 d on the head of the robot 1 may be the second image capturing unit 2 .
In the example shown in Fig. 2, the hand 14 has five fingers, but the number of fingers may be two or more. Also, in Fig. 2, an example of a robot with two arms is shown, but the robot may have one arm. The number of image capturing units is merely an example and is not limited to this. The image capturing unit may be at least one of the palm, fingertips, and wrist, or multiple units may be provided on the hand 14.

[Data format to be acquired]
Next, an example of the format of data acquired by the robot remote operation control device 5 will be described.
3 is a diagram showing an example of a data format of joint angle information according to this embodiment. As shown in FIG. 3, a timestamp is associated with the joint angle information.
4 is a diagram showing an example of the data format of image information according to this embodiment. As shown in FIG. 4, a timestamp is associated with the image information.
The joint angle information and image information are stored, for example, temporarily, in the storage unit 56 .

[Example of processing performed by the manipulated variable determination unit]
Next, an example of the processing performed by the operation amount determiner 52 will be described.
The operation amount determination unit 52 determines the operation amount of the robot 1 based on the operator state information acquired by the operator information acquisition unit 513. Note that the operation amount determination unit 52 may, for example, estimate the target object, its position, its shape, etc., using the image acquired by the image acquisition unit 511, and determine the operation amount of the robot 1 using the estimated information about the target object as well. Furthermore, the operation amount determination unit 52 may estimate taxonomy information (see, for example, Reference 1) related to the task based on the operator state information acquired by the operator information acquisition unit 513, and determine the operation amount of the robot 1 using the estimated taxonomy information as well.

Reference 1: Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77

[Example of processing performed by the robot posture calculation unit]
Next, an example of the processing performed by the robot posture calculation unit 53 will be described.
The robot posture calculation unit 53 calculates the posture of the robot 1, for example, at predetermined time intervals, using the joint angle information (including timestamps) acquired by the sensor information acquisition unit 512 and the three-dimensional shape model stored in the robot model storage unit 6. Note that the robot posture calculation unit 53 calculates the posture of the arm 13 and the end effector 14 depending on the task, and does not necessarily need to calculate the posture of the body 16, head 17, etc.

[Example of processing performed by the spatial reproduction image rendering unit]
Next, an example of the processing performed by the spatial reproduction image drawing unit 54 will be described.
5 is a diagram illustrating an example of processing performed by the spatial reproduction image rendering unit according to this embodiment. The spatial reproduction image rendering unit 54 renders, as a spatial reproduction image, an image in which spatial occlusion by the body of the robot 1 has been eliminated, based on the spatial information and the robot posture information. More specifically, the spatial reproduction image rendering unit 54 first calculates how the robot appears from the operator's viewpoint. If the calculated appearance of the robot from the operator's viewpoint obstructs the operator's view, the spatial reproduction image rendering unit 54 uses the acquired image to convert it into an image in which the view is not obstructed, and presents it.

The spatial reproduction image drawing unit 54 may be provided with a trained model. Fig. 6 is a diagram showing an example of input and output of a model provided in the spatial reproduction image drawing unit according to this embodiment.
The model 541 may be trained by inputting image information, joint angle information, and robot model information in advance, and outputting a provided image using the provided image as training data. The spatial reproduction image rendering unit 54 may input image information, joint angle information, and robot model information to the trained model during operation, and output a provided image. The model may be configured using a network such as a deep neural network (DNN) or a recurrent neural network (RNN).

[Example of processing procedure]
Next, a description will be given of an example of a processing procedure performed by the robot remote operation control system 9. Fig. 7 is a flowchart of a processing procedure performed by the robot remote operation control system according to this embodiment.

(Step S1) The image acquisition unit 511 acquires the image captured by the first image capture unit 11 and the image captured by the second image capture unit 2.

(Step S2) The sensor information acquisition unit 512 acquires sensor information (including joint angle information) detected by the sensor 12.

(Step S3) The operator information acquisition unit 513 acquires operator status information from the HMD 3 and the operation detection unit 4.

(Step S4) The robot posture calculation unit 53 calculates the posture of the robot 1 based on the information acquired by the sensor information acquisition unit 512. Note that the robot posture calculation unit 53 may also calculate the posture of the robot 1 based on operator status information. Alternatively, the robot posture calculation unit 53 may calculate the posture of the robot 1 based on the information acquired by the sensor information acquisition unit 512 and the operator status information.

(Step S5) The spatial reproduction image rendering unit 54 uses the acquired information and the three-dimensional shape model of the robot to calculate how the robot will appear from the operator's viewpoint.

(Step S6) The spatial reproduction image drawing unit 54 determines whether the body of the robot 1 (arm 13, end effector 14, head 17, body, etc.) is blocking the operator's field of view in the calculated view of the robot from the operator's viewpoint. If the body of the robot 1 is blocking the creator's field of view (Step S6; YES), the spatial reproduction image drawing unit 54 proceeds to processing of Step S7. If the body of the robot 1 is not blocking the creator's field of view (Step S6; NO), the spatial reproduction image drawing unit 54 proceeds to processing of Step S9.

(Step S7) The spatial reproduction image rendering unit 54 converts the acquired image into an image from the operator's viewpoint. Note that the spatial reproduction image rendering unit 54 may convert the acquired image into an image from the operator's viewpoint, for example, using a trained model.

(Step S8) The spatial reproduction image rendering unit 54 synthesizes an image from the operator's viewpoint so as to eliminate occlusion by the robot 1's body by synthesizing an image converted from the acquired image into an image of how it would appear from the operator's viewpoint.

(Step S9) If the body of the robot 1 does not obstruct the operator's field of view, the spatial reproduction image drawing unit 54 provides the operator with an image of how it appears from the operator's viewpoint, generated based on the acquired image, via the image output unit 57. If the body of the robot 1 obstructs the operator's field of view, the spatial reproduction image drawing unit 54 provides the operator with an image of how it appears from the operator's viewpoint, generated based on the acquired image and converted into an image of how it appears from the operator's viewpoint, via the image output unit 57.

Note that the operational procedure shown in Figure 7 is an example and is not limited to this. For example, the processing of steps S1 to S3 does not have to be in this order, or may be performed simultaneously in parallel.

[Method for determining whether the operator's field of vision is obstructed]
Here, an example of a method for determining whether or not the operator's field of vision is obstructed will be described.
The robot remote operation control device 5 can grasp the position and state of the target object from the captured image before grasping the target object. The robot remote operation control device 5 can also grasp the posture of the robot from the information acquired by the sensor information acquisition unit 512 and the three-dimensional model of the robot 1 stored in the robot model storage unit 6.

The robot remote operation control device 5 uses this information, a three-dimensional model of the robot 1, and the captured image to create an image of the operator's field of view. Based on information on the position of the target object and the position and orientation of the robot 1's arm 13 and end effector 14, the robot remote operation control device 5 determines whether the body of the robot 1 is blocking the operator's field of view in the generated image of the operator's field of view.

Figure 8 shows an example image of a state in which the arm of a robot blocks the line of sight of an operator. Figure 8 is an example of an image drawn using a three-dimensional model of the robot, showing the arm and hand of robot 1 performing a task. For example, this is an example of a state in which the arm or hand blocks the line of sight of the operator by blocking the target object.

Figure 9 is an image diagram showing an example of an image in which the robot's arm is blocking the operator's line of sight. In the example of Figure 9, the target object obj is blocked by the right hand 14R. Even if an image in which the operator's field of view is blocked is provided, the operator will be unable to issue a grasp operation instruction, or will find it difficult to issue a grasp operation instruction, because the target object obj is blocked.

For this reason, in this embodiment, when the operator's line of sight is blocked in this way, an image is synthesized based on the acquired image, converted by the generated robot 1 into an image that appears from the operator's viewpoint so as to erase the blocked portion. Figure 10 is an illustration of an example of an image synthesized to erase the blocked portion. As shown in Figure 10, the robot remote operation control device 5 generates an image in which the portion of the target object obj that is blocked by a palm or the like is not blocked, using the acquired image and a three-dimensional model of the robot 1. Note that the robot remote operation control device 5 may, for example, generate an image in which the blocking palm portion is made semi-transparent so that the target object obj can be seen.

As described above, in this embodiment, detailed three-dimensional shape information of the known robot 1, acquired time-stamped information on the angles of each joint of the robot 1, and acquired time-stamped images are used to generate and synthesize an image in which any obstructions to the operator's line of sight have been removed.

As a result, according to this embodiment, by eliminating the obstruction caused by the robot's body, the operator's sense of discomfort when operating the robot can be reduced.

Furthermore, because the body shape of the robot 1 is different from that of the operator, in images captured by the first image capture unit 11 mounted on the head of the robot 1, the view is obstructed by the arms of the robot 1, making operation feel strange. This embodiment can reduce this sense of strangeness.

[Modification]
In the above example, an image from the operator's viewpoint is provided, but the image to be provided is not limited to this. Since the robot remote operation control device 5 already knows the position and orientation information of the image capturing units (first image capturing unit 11, second image capturing unit 2), it is also possible to synthesize and provide an image from any viewpoint using the acquired information. For example, the robot remote operation control device 5 can also provide an image of the robot 1 with its body hidden, as shown in FIG. 11 .

Figure 11 shows an example of an image with the robot's body removed. In the example of Figure 11, for example, several objects placed on a table and the arms other than the fingers on the right hand and the gripper on the left hand have been removed.

As described above, in this modified example, an image from an arbitrary viewpoint is generated using detailed three-dimensional shape information of the known robot 1, acquired time-stamped information on the angles of each joint of the robot 1, and acquired time-stamped images.

Note that an image from an arbitrary viewpoint may be provided, for example, by the operator operating the HMD 3 or the line of sight detection results of the operation detection unit 4 based on the operation.

In addition, a program for implementing all or part of the functions of the robot remote operation control device 5 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the robot remote operation control device 5. Note that the term "computer system" as used here includes hardware such as an OS and peripheral devices. It also includes a WWW system equipped with a homepage provision environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

The above program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium, or by transmission waves within the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded on the computer system.

The above describes the form for carrying out the present invention using an embodiment, but the present invention is in no way limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

9...Robot remote operation control system, 1...Robot, 2...Second image capture unit, 3...HMD, 4...Operation detection unit, 5...Robot remote operation control device, 6...Robot model memory unit, 11...First image capture unit, 12...Sensor, 13...Arm, 14...End effector, 15...Drive unit, 31...Image display unit, 32...Gaze detection unit, 51...Acquisition unit, 52...Operation amount determination unit, 53...Robot posture calculation unit, 54...Spatial reproduction image drawing unit, 55...Control unit, 56...Memory unit, 511...Image acquisition unit, 512...Sensor information acquisition unit, 513...Operator information acquisition unit, 541...Model

Claims (8)

In robot remote control, an operator remotely controls a robot.
a spatial information sensor for acquiring spatial information of a space in which the robot exists;
an image acquisition unit that acquires an image captured by an imaging unit that captures an image of a space where the robot exists;
an image output unit that presents to the operator a spatial reproduction image in which the space in which the robot exists is reproduced from the spatial information;
a robot posture calculation unit that calculates a posture of the robot based on the three-dimensional model of the robot and the spatial information;
a spatial reproduction image drawing unit that calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, determines whether the body of the robot is blocking the operator's field of view based on the appearance of the robot, removes the spatial occlusion caused by the body of the robot when it is determined that the field of view is blocked , and draws an image converted from the image acquired by the image acquisition unit as a spatial reproduction image ,
the spatial reproduction image rendering unit generates an image in which the parts of the robot's body that are blocking the view are semi-transparent.
Robot remote control device. the spatial reproduction image drawing unit calculates how the robot appears from the operator's viewpoint based on the spatial information and the information calculated by the robot posture calculation unit, determines whether the body of the robot is blocking the operator's field of view, and, if the body of the robot is blocking the operator's field of view, draws an image from an arbitrary viewpoint of the operator as a spatial reproduction image.
The robot remote control device according to claim 1 . an information acquisition unit that acquires operator status information about a status of an operator operating the robot;
the robot posture calculation unit calculates the posture of the robot based on the three-dimensional model of the robot, the spatial information sensor, and the operator state information;
The robot remote control device according to claim 1 or 2 . The spatial information sensor has a known position, acquires data in a time series, and assigns a time stamp to the data.
The robot remote control device according to claim 1 . the spatial reproduction image rendering unit calculates how the robot appears from the operator's viewpoint using a model trained using the spatial information and an image from the operator's viewpoint, which is training data;
The robot remote control device according to claim 1 . further comprising an operation amount determination unit that determines an operation amount of the robot based on the operator state information;
The robot remote control device according to claim 3 . A robot remote control control device that allows an operator to remotely control a robot,
acquiring spatial information of the space in which the robot exists;
acquiring an image captured by an imaging unit that captures an image of a space in which the robot exists;
calculating a posture of the robot based on the three-dimensional model of the robot and the spatial information;
From the spatial information and the calculated information of the posture of the robot, how the robot appears from the viewpoint of the operator is calculated, and based on how the robot appears, it is determined whether the body of the robot is blocking the field of view of the operator, and if it is determined that the field of view is blocked, the spatial occlusion caused by the body of the robot is eliminated and converted into an image captured by the imaging unit, and further an image is generated in which the part of the body of the robot that is blocking the field of view is made semi-transparent, and the image is rendered as a spatial reproduction image and presented to the operator.
A method for remotely controlling a robot. The computer of the robot remote control control device, which allows the operator to remotely control the robot,
acquiring spatial information of a space in which the robot exists;
acquiring an image captured by an imaging unit that captures an image of a space in which the robot exists;
calculating a posture of the robot based on the three-dimensional model of the robot and the spatial information;
From the spatial information and the calculated information of the posture of the robot, how the robot appears from the viewpoint of the operator is calculated, and based on how the robot appears, it is determined whether the body of the robot is blocking the field of view of the operator, and if it is determined that the field of view is blocked, the spatial occlusion caused by the body of the robot is eliminated and converted into an image captured by the imaging unit, and further an image is generated in which the part of the body of the robot that is blocking the field of view is made semi-transparent, and the image is rendered as a spatial reproduction image and presented to the operator.
program.