JP7809033B2 - Remote operation assistance system, remote operation assistance method, and program - Google Patents

Description

The present invention relates to a remote operation assistance system, a remote operation assistance method, and a program.

A control device has been proposed that allows a robot to grasp an object. One such control device involves tentatively determining a physically contactable grasp point, determining a grasp force taking into account the balance of forces, evaluating the quality of the grasp force as the volume of the envelope of the grasp force, and then searching for a grasp point of higher quality (see, for example, Patent Document 1).

Patent No. 6476358

Technologies such as those described in Patent Document 1 register the shape and gripping point of each object in advance, estimate the object's position and orientation using the object's shape, and then generate a gripping trajectory based on the relationship between the robot's current orientation and the gripping point. For this reason, technologies such as those described in Patent Document 1 have the problem of only being able to handle objects (shapes and weights) that have been registered in the system in advance.

The present invention was made in consideration of the above-mentioned problems, and aims to provide a remote operation assistance system, remote operation assistance method, and program that can perform work on objects that have not been registered in the system in advance.

(1) In order to achieve the above object, one aspect of the present invention provides a remote operation assistance system for remotely operating at least an end effector, comprising: a motion acquisition unit that acquires information regarding the motion of an operator operating the end effector; an intention understanding unit that uses the information acquired by the motion acquisition unit to estimate a target object to be operated by the end effector and a task that is a method of operating the target object; an environmental situation determination unit that acquires environmental information about the environment in which the end effector is operated; and a manipulation amount determination unit that acquires information from the intention understanding unit and the environmental situation determination unit and determines a manipulation amount for the end effector from the acquired information, wherein the manipulation amount determination unit comprises an approximation result reflection unit that approximates the object shape of the target object to geometric primitives and compares the approximation result with registered information of a pre-registered manipulation strategy for the end effector and reflects the approximated result.

(2) In the remote operation assistance system according to (1) of one aspect of the present invention, the approximation result reflecting unit performs shape fitting of the geometric primitives for each instance ID (identifier) of the target object, where the environment information is shape information related to the target object including an instance ID. Note that in this embodiment, the shape related to the target object is voxel information or point cloud information.

(3) In the remote operation assistance system according to (1) or (2) of one aspect of the present invention, the geometric basic element is one, multiple, or one of a region of interest.

(4) Furthermore, in the remote operation assistance system described in (3) according to one aspect of the present invention, when the geometric primitive is the region of interest, the approximation result reflecting unit transforms the average shape of the target object into the geometric primitive and calculates a mapping function between regions, obtains a grasping probability map based on the motion and taxonomy estimated by the intention understanding unit and the class ID separated by the environmental situation determining unit, maps the grasping probability map to the region of the geometric primitive based on the calculated mapping function of correspondence between regions, and solves an optimization problem to find a grasping point that maximizes the probability of the shortest path and the shortest route while satisfying geometric constraints from the current wrist posture.

(5) In addition, in the remote operation assistance system according to any one of (1) to (4) according to one aspect of the present invention, the environmental situation determination unit includes an RGB sensor that acquires RGB data, a depth sensor that acquires a depth image, and an instance segmentation (Instance the mask generation unit uses the color image output by the RGB sensor to generate a mask image based on the instance ID of an object in the image using a method of image segmentation; an adder unit uses the depth image and the mask image to generate a depth image for each instance ID; a self-position estimation unit performs image processing on the color image output by the RGB sensor to detect the position of the environmental situation assessment unit; a three-dimensional reconstruction unit uses the posture information of the environmental situation assessment unit output by the self-position estimation unit and the depth image for each instance ID for each class output by the adder unit to perform three-dimensional reconstruction to obtain a point cloud for each instance ID for each class; and a data integration unit acquires a point cloud for each instance ID for each key frame, acquires and manages the current data at time t and time series data for past frames t-1, t-2, ..., and integrates this data.

(6) To achieve the above object, one aspect of the present invention provides a remote operation assistance method in a remote operation assistance system that remotely operates at least an end effector, the remote operation assistance method including: a motion acquisition step in which a motion acquisition unit acquires information about at least the motion of an operator operating the end effector; an intention understanding step in which an intention understanding unit uses the information acquired in the motion acquisition step to infer a target object to be operated by the end effector and a task that is a method of operating the target object; an environmental situation determination step in which an environmental situation determination unit acquires environmental information about the environment in which the end effector is operated; and a manipulation amount determination step in which a manipulation amount determination unit acquires the information acquired in the intention understanding step and the information acquired in the environmental situation determination step and determines a manipulation amount for the end effector from the acquired information; and in the manipulation amount determination step, the manipulation amount determination unit approximates the object shape of the target object to geometric primitives and compares the approximation result with registered information of a pre-registered manipulation strategy for the end effector to reflect the approximated result.

(7) In order to achieve the above object, one aspect of the present invention provides a program for a computer in a remote operation assistance system that remotely operates at least an end effector, the program executing the following: acquiring motion information related to the motion of an operator operating the end effector; using the motion information, estimating a target object to be operated by the end effector and a task that is a method of operating the target object; acquiring environmental information about the environment in which the end effector is operated; acquiring information about the estimated target object and task and the environmental information; determining an operation amount for the end effector from the acquired information; approximating the object shape of the target object to geometric primitives in determining the operation amount for the end effector; and comparing the approximation result with registered information of a pre-registered operation strategy for the end effector and reflecting the result in the calculation.

By using (1) to (7), it is possible to work with objects that have not been registered in the system in advance.

1 is a diagram illustrating a configuration example of a remote operation assistance system according to an embodiment; FIG. 2 is a diagram illustrating an example of the configuration of an environmental situation determination unit according to the embodiment. FIG. 1 is a diagram illustrating an overview of remote control according to an embodiment. FIG. 10 is a diagram illustrating an example of a geometric primitive. 10 is a flowchart of an example of a processing procedure performed by the remote operation assistance system according to the embodiment. 10 is a flowchart of a processing procedure for a single primitive and multiple primitives according to an embodiment. 1 is a diagram for explaining an outline of a configuration and an outline of a process of a remote operation assistance system according to an embodiment; FIG. 10 is a diagram for explaining a comparative example.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized.
In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted.
Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[Configuration example of remote operation assistance system]
1 is a diagram showing an example of the configuration of a remote operation assistance system according to this embodiment. As shown in FIG. 1, the remote operation assistance system 1 includes an environmental situation determination unit 2 (2-1, 2-2, 2-3, ...), an HMD 3, an instruction detection unit 4, an action acquisition unit 5, an intention understanding unit 6, an operation amount determination unit 7, a robot 15, and a control unit 16.
The HMD 3 includes, for example, a gaze detection unit 31 .
The operation amount determination unit 7 includes, for example, a model 8, a data integration unit 9, an approximation result reflection unit 17, a motion estimation unit 11, a grip DB 12, and a database 14.
The approximation result reflecting unit 17 includes a shape approximation unit 10 and a grasp planning unit 13 .
The robot 15 includes, for example, an end effector 151 , an arm 152 , and a sensor 153 .

The environmental situation determination unit 2 is equipped with an RGB sensor and a depth sensor, as described below. The environmental situation determination unit 2 acquires environmental information about the environment in which the end effector 151 is operated. The environmental situation determination unit 2 outputs shape information about the target object. Note that in this embodiment, the shape about the target object is, for example, voxel information or point cloud information. Furthermore, multiple environmental situation determination units 2 are installed in the operating environment, for example. Note that an example configuration of the environmental situation determination unit 2 will be described later.

The HMD 3 is, for example, a head-mounted display device. The HMD 3 may be, for example, a glasses-type device, and may be for one eye or both eyes. The HMD 3 also includes a display device, and may acquire and display images captured by the RGB sensor 211 (Figure 2) of the environmental condition assessment unit 2, or may display virtual images generated using the acquired image data.

The gaze detection unit 31, for example, captures an image of the operator's eyes and performs image processing on the captured image to detect the gaze.

The instruction detection unit 4 detects the operator's finger joint angles and wrist posture. The instruction detection unit 4 is, for example, a data glove, a joint angle detection sensor, etc.

The motion acquisition unit 5 acquires information about at least the motion of the operator operating the end effector 151. The motion acquisition unit 5 acquires, for example, the detection results detected by the gaze detection unit 31 and the detection results detected by the instruction detection unit 4.

The intention understanding unit 6 uses the information acquired by the action acquisition unit 5 to estimate the target object to be operated by the end effector 151 and the task (operation intention), which is the method of operating the target object.

The operation amount determination unit 7 acquires information output by the intention understanding unit 6 and information output by the environmental situation determination unit 2, and determines the operation amount of the end effector 151 from the acquired information.

Model 8 is a trained model used by the motion estimation unit 11 when estimating motion.

The data integration unit 9 generates integrated information of the three-dimensional shape using voxel information contained in the shape information about the target object output by multiple environmental situation assessment units 2. The data integration unit 9 may also generate three-dimensional information by integrating point cloud information contained in the shape information about the target object output by multiple environmental situation assessment units 2. Since the integrated information includes an instance ID, the data integration unit 9 outputs integrated information of the three-dimensional shape integrated for each instance ID.

The approximation result reflection unit 17 compares the approximation result with the registration information of the pre-registered operation strategy of the end effector 151 and reflects it.

The shape approximation unit 10 acquires the integrated information of the integrated three-dimensional shape output by the data integration unit 9. The shape approximation unit 10 generates geometric primitive information by performing shape fitting processing on geometric primitives for each instance ID. As described below, the shape approximation unit 10 may perform shape approximation using a single primitive, multiple primitives, or a region of interest. For multiple primitives, geometric information is used as information for classifying shapes. When performing shape approximation using multiple primitives, the shape approximation unit 10 acquires the affordance of each region by referencing the database 14 after fitting each primitive. The shape approximation unit 10 classifies the objects, for example, into object classes (e.g., plastic bottle, wrench, box, etc.).

The motion estimation unit 11 acquires operation intention information from the intention understanding unit 6, joint angle measurements from the robot 15, and instance IDs (identification information) and geometric primitives of visible objects (detected by the environmental situation assessment unit 2) from the shape approximation unit 10. The motion estimation unit 11 references the model 8 to estimate a motion, which is a set of operation intentions, from the operation intention, robot state, and possible operations associated with the target object. For example, the motion of switching hands to hold a wrench is composed of operation intentions for right-hand grip, two-hand support, and left-hand grip. For example, the motion estimation unit 11 estimates the next motion the robot 15 should perform. The motion estimation unit 11 outputs estimated motion information for the robot 15's hands. The motion information also includes taxonomy information related to the task (see, for example, Reference 1). To reduce the amount of data, the model 8 does not store a wide variety of plastic bottle models, but instead stores a representative model of a plastic bottle. Therefore, the movement estimation unit 11 can estimate movement by using the class of the target object and the information stored in the model 8.

Reference 1: Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77

The grasping DB12 is a database that stores shape models with probabilities, and holds them for each class ID (identifier). Shape refers to the average shape of the class. Probability refers to the grasping probability for each operation intention, and is held for each polygon, for example. This shape is transformed to match the geometric primitive shape to obtain the grasping probability for the obtained instance shape, but this may also be approximated using a neural network. The grasping DB12 stores the general weight (per unit volume) of the object class. The grasping DB12 stores the grasping plan for the average shape for each taxonomy and class.

The grasp planning unit 13 acquires geometric primitive shape information for each instance ID from the shape approximation unit 10 and motion information estimated from the motion estimation unit 11. The grasp planning unit 13 determines a grasp point (grasp position) based on the geometric primitive for each instance ID included in the acquired information, the future operation intention and taxonomy obtained from the motion, and the grasp probability for each operation intention. For example, the grasp plan keeps the fingers closed until contact occurs after the grasp point is determined. The grasp planning unit 13 outputs hand trajectory sequence information and finger trajectory sequence information based on the grasp point. The processing performed by the grasp planning unit 13 will be described later.

Database 14 stores, for example, the ahordances possessed by primitives. In the case of multiple primitives, database 14 stores the ahordances possessed by each primitive. Database 14 associates and stores instance IDs with the class IDs to which they belong. Database 14 stores average shape models for each class. For primitives in the region of interest, database 14 stores positional information within the region that is easy to grasp, associated with the average shape.

The robot 15 is, for example, one of a single-arm robot, a double-arm robot, and a multi-arm robot.
The end effector 151 has at least three fingers. The number of fingers included in the end effector 151 may be any number that allows the taxonomy to be realized.
The arm 152 has an end effector 151 connected to its tip.

The control unit 16 generates joint angle command values for the end effector 151 and arm 152 based on the hand trajectory information and finger trajectory information output by the grasp planning unit 13 and the joint angle measurement values output by the sensor 153 of the robot 15, and controls the operation of the robot 15.

[Configuration example of environmental situation determination unit]
Next, an example of the configuration of the environmental situation determination unit 2 will be described.
2 is a diagram showing an example of the configuration of the environmental situation determination unit according to this embodiment. As shown in FIG. 2, the environmental situation determination unit 2 includes, for example, an environmental sensor 21, a self-position estimation unit 22, a mask generation unit 23, an addition unit 24, a 3D reconstruction unit 25, and a data integration unit 26.
The environmental sensor 21 includes, for example, an RGB sensor 211 and a depth sensor 212 .

The RGB sensor 211 is an imaging device such as a CCD (Charge Coupled Device) imaging device or a CMOS (Complementary Metal Oxide Semiconductor) imaging device, and outputs an RGB (red, green, blue) color image. The RGB sensor 211 also includes, for example, a fisheye lens.

The depth sensor 212 acquires a depth image including depth information and outputs the acquired depth image. Note that the depth image also includes brightness information as seen by the depth sensor 212.

The adder 24 generates a depth image for each instance ID using the depth image and mask image, and outputs the generated depth image for each instance ID. The output image contains the depth for each instance ID. This makes it possible to extract the locations of areas with the same instance ID, and therefore the distance for each instance ID.

The mask generation unit 23 uses the color image output by the RGB sensor 211, for example, using a technique called instance segmentation, to generate and output a mask image based on the instance ID (identification information) of an object in the image. Instance segmentation is the problem of estimating the foreground region mask of an object instance appearing in an image or RGB-D image, while distinguishing each object instance from other objects. The foreground region mask represents "foreground object or background" using, for example, two values, [1 or 0].

The self-position estimation unit 22 performs image processing on the color image output by the RGB sensor 211 to detect the position of the environmental situation assessment unit 2 and output attitude information of the image capture device.

The three-dimensional reconstruction unit 25 uses the orientation information of the image capture device output by the self-position estimation unit 22 and the depth image for each instance ID for each class output by the addition unit 24 to perform three-dimensional reconstruction to obtain a point cloud for each instance ID for each class. In other words, the three-dimensional reconstruction unit 25 divides the instance IDs for each class and constructs the shape of the target object using, for example, voxels or a point cloud. Note that a point cloud is point cloud data, and is three-dimensional data that contains information such as basic X, Y, and Z position information and color.

The data integration unit 26 acquires a point cloud for each instance ID for each key frame. The data integration unit 26 also acquires and manages the current data at time t and time series data for past frames such as t-1 and t-2, and integrates this data. The data integration unit 26 outputs shape information about the target object, including the current frame and past frames, to the operation amount determination unit 7. For example, if two objects are placed on a table, an instance ID is associated with each of the two classes (first object, second object).

[Remote control overview]
Fig. 3 is a diagram showing an overview of remote control according to this embodiment. Note that, although the robot 15 shown in Fig. 3 is an example having a pair of arms, a head, and a body, the configuration and shape of the robot 15 are not limited to this.
3 , the operator Us wears, for example, an HMD (head-mounted display) 3 and instruction detection units 4a and 4b. An environmental sensor 21a is installed on the robot 15 or around the robot 15, and an environmental sensor 21b is also installed in the work environment. The environmental sensor 21a may be attached to the robot 15. The robot 15 also includes an end effector 151 (151a, 151b) and an arm 152. The operator Us remotely controls the robot 15 by moving the hand or fingers wearing the instruction detection units 4a and 4b while viewing the image displayed on the HMD 3.

[Example of geometric primitives]
Here, an example of a geometric primitive will be described using an example of a task in which the robot 15 is made to grip a wrench and tighten a screw.
4 is a diagram for explaining an example of a geometric primitive. In a task in which the robot 15 is made to hold a wrench and tighten a screw, the object of the operator's attention is the wrench.

In the example of image g100, the target object is a single primitive with one frame (g101).

In the example of image g110, the target object is a multiple primitive with two frames (g111, g112).

In the image g120, in the case of screw tightening work, the area of interest is within the frame g112.
Thus, shape fitting in geometric primitives may be performed on a single frame, on two or more frames, or using regions of interest.

In this way, in this embodiment, the target object is approximated, for example, on a plane, using geometric primitives. Note that in this embodiment, geometric primitives are basic shapes in Euclidean geometry, such as triangles, squares, rectangles, trapezoids, polygons, circles, and ellipses.

[Example of processing procedure]
Next, a description will be given of an example of a processing procedure performed by the remote operation assistance system 1. Fig. 5 is a flowchart of an example of a processing procedure performed by the remote operation assistance system 1 according to this embodiment.

(Step S1) The environmental situation determination unit 2 acquires environmental information about the environment in which the end effector 151 is operated. The environmental situation determination unit 2 outputs the acquired environmental information, which is shape information about the target object, to the operation amount determination unit 7.

(Step S2) The data integration unit 9 integrates three-dimensional shapes for each instance ID using voxel information contained in the shape information related to the target object acquired from multiple environmental situation assessment units 2. The shape approximation unit 10 generates geometric primitive information based on the integrated information of the integrated three-dimensional shapes for each instance ID. The grasp planning unit 13 acquires the geometric primitive information for each instance ID from the shape approximation unit 10.

(Step S3) The operation amount determination unit 7 acquires information about the movement of the operator operating the end effector 151 from the HMD 3 and the instruction detection unit 4.

(Step S4) The intention understanding unit 6 uses the information output by the action acquisition unit 5 to estimate the target object to be operated by the end effector 151 and the task, which is the method for operating the target object.

(Step S5) The motion estimation unit 11 estimates a motion, which is a collection of operational intentions, from the operational intentions, the robot state, and possible operations linked to the target object. The grasp planning unit 13 acquires motion information from the motion estimation unit 11.

(Step S6) The grasp planning unit 13 uses the geometric primitive information and estimated motion information for each acquired instance ID to search the database 14 for the class ID to which the instance ID belongs. For example, if there are instances called Spanner A and Spanner B in the Spanner class, the grasp planning unit 13 searches the database 14 for these class IDs.

(Step S7) The grasp planning unit 13 obtains the average shape model associated with the class of the class ID from the database 14. For example, if the class is a wrench, the average shape model is a model of the average shape of a wrench. Note that such average shape models for each class are stored in the database 14.

(Step S8) The grasp planning unit 13 transforms the average shape into a geometric primitive to obtain a mapping function between regions. The grasp planning unit 13 can obtain a mapping function between the average shape and the primitive shape, for example, by fitting the geometric primitive to shape information about the target object. Note that the target object is not limited to a rigid body; it may be a non-rigid object as long as its deformation can be defined. The grasp planning unit 13 also obtains the general weight (per unit volume) of the object class from the database 14.

(Step S9) The grasping planning unit 13 obtains a grasping probability map from the grasping DB 12 based on the motion, taxonomy, and class ID.

(Step S10) The grasp planning unit 13 maps the grasp probability map to the geometric primitive region based on the obtained mapping function of correspondence between regions. This makes it possible to obtain probability heat map-like information corresponding to the geometric primitive shape obtained online.

(Step S11) The grasp planning unit 13 solves an optimization problem to find the grasp point that has the shortest path and the highest probability of being the shortest route, while satisfying geometric constraints, based on the current wrist posture.

(Step S12) After a gripping point is selected, the grasp planning unit 13 calculates a trajectory for approaching the gripping point and outputs it to the robot 15. The grasp planning unit 13 adaptively controls the change in weight from the general weight (per unit volume) for the object class.

Note that in the above-described embodiment, grasping was used as an example of a task, but the task content is not limited to this.

[Processing for single and multiple primitives]
Here, the processing for the single primitive and multiple primitives will be further explained. Fig. 6 is a flowchart showing the processing procedure for the single primitive and multiple primitives according to this embodiment.

(Step S101) The environmental situation determination unit 2 acquires environmental information about the environment in which the end effector 151 is operated. The environmental situation determination unit 2 outputs the acquired environmental information, which is shape information about the target object, to the operation amount determination unit 7.

(Step S102) The data integration unit 9 integrates three-dimensional shapes for each instance ID using voxel information contained in the shape information related to the target object acquired from multiple environmental situation assessment units 2. The shape approximation unit 10 generates geometric primitive information based on the integrated information of the integrated three-dimensional shapes for each instance ID. The grasp planning unit 13 acquires the geometric primitive information for each instance ID from the shape approximation unit 10.

(Step S103) The operation amount determination unit 7 acquires information about the movement of the operator operating the end effector 151 from the HMD 3 and the instruction detection unit 4.

(Step S104) The intention understanding unit 6 uses the information output by the action acquisition unit 5 to estimate the target object to be operated by the end effector and the task, which is the method for operating the target object.

(Step S105) The motion estimation unit 11 estimates a motion, which is a collection of operational intentions, from the operational intention, robot state, and possible operations linked to the target object. The grasp planning unit 13 acquires motion information from the motion estimation unit 11.

(Step S106) The grasp planning unit 13 uses the acquired geometric primitive information and estimated motion information to search the database 14 for the class ID to which the instance ID belongs. For example, if there are instances called Spanner A and Spanner B in the Spanner class, the database 14 is searched for these class IDs.

(Step S107) In the case of multiple primitives, the grasp planning unit 13 determines the selectable taxonomy for each primitive based on the operator's input, the class ID, and the primitive's operation. Note that in the case of a single primitive, the grasp planning unit 13 determines the taxonomy based on the operator's input.

(Step S108) The grasp planning unit 13 determines the initial weight of the object based on the density corresponding to the class ID and the estimated volume.

(Step S109) The grasp planning unit 13 solves an optimization problem to select a grasp point from the current wrist posture that satisfies the geometric constraints and reaches the object surface via the shortest path.

(Step S110) After a gripping point is selected, the grasp planning unit 13 calculates a trajectory for approaching the gripping point and outputs it to the robot 15. The grasp planning unit 13 adaptively controls the change in weight from the general weight (per unit volume) for the object class.

[Processing and configuration overview]
Here, an outline of the configuration and processing of the remote operation assistance system 1 will be described.
FIG. 7 is a diagram for explaining an outline of the configuration and processing of the remote operation assistance system according to this embodiment.
7, the mask generation unit 23 generates and outputs a mask image based on the instance ID of an object in an image in RGB data. Note that a prerequisite here is that the objects in the image can be classified into classes.
The three-dimensional reconstruction unit 25 obtains a point cloud for each instance ID for each class by three-dimensional reconstruction based on the mask image and the depth image. The three-dimensional reconstruction unit 25 clusters and outputs voxels or a point group (point cloud), which is shape information related to the target object. The three-dimensional reconstruction unit 25 performs active sensing using the RGB sensor 211 and the depth sensor 212, and integrates the sensed information into three-dimensional points.
The shape approximation unit 10 performs primitive fitting using shape information on the clustered target object to approximate the object shape with geometric basic elements (for example, a rectangular parallelepiped, a cylinder, a sphere, etc.).
The grasp planning unit 13 realizes object grasping according to the operation intention by scaling a pre-registered grasp strategy.

[Comparative Example]
A comparative example will now be described with reference to Fig. 8. Fig. 8 is a diagram for explaining the comparative example.
As shown in Figure 8, conventional technology could only handle objects whose information was registered in advance in a database. For example, even if the object was a wrench, if the shape and weight were different, it had to be registered. Furthermore, conventional technology required that the three-dimensional shape and weight of the target object be known. Furthermore, with congestion technology, the gripping point and gripping posture of the target object also had to be known.

In contrast, in this embodiment, only the object class (e.g., a wrench) and a grasping strategy corresponding to the operation intention are provided in advance as prerequisites. Then, in this embodiment, the object shape for which a grasping strategy corresponding to the object class and operation intention is provided in advance is approximated with a geometric primitive (e.g., a rectangular parallelepiped), and the pre-registered grasping strategy is scaled to achieve object grasping corresponding to the operation intention. Furthermore, in this embodiment, the weight is adaptively controlled by the amount of change from the general weight (per unit volume) of the object class.

As a result, according to this embodiment, less information needs to be registered in advance, allowing for grasp planning for a variety of instances.

In addition, a program for realizing all or part of the functions of the manipulated variable determination unit 7 in this invention may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the manipulated variable determination unit 7. Note that the term "computer system" as used here includes hardware such as an OS and peripheral devices. It also includes a WWW system equipped with a homepage provision environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

The above program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium, or by transmission waves within the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded on the computer system.

The above describes the form for carrying out the present invention using an embodiment, but the present invention is in no way limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

1... Remote operation assistance system, 2, 2-1, 2-2, 2-3... Environmental situation judgment unit, 3... HMD, 4... Instruction detection unit, 5... Motion acquisition unit, 6... Intention understanding unit, 7... Operation amount determination unit, 8... Model, 9... Data integration unit, 10... Shape approximation unit, 11... Motion estimation unit, 12... Grasp DB, 13... Grasp planning unit, 14... Database, 15... Robot, 16... Control unit, 17... Approximation result reflection unit, 31... Gaze detection unit, 151... End effector, 152... Arm, 153... Sensor, 21... Environmental sensor, 22... Self-position estimation unit, 23... Mask generation unit, 24... Addition unit, 25... 3D reconstruction unit, 26... Data integration unit, 211... RGB sensor, 212... Depth sensor

Claims (7)

A teleoperation assistance system for remotely operating at least an end effector,
a motion acquisition unit that acquires motion information relating to a motion of an operator operating at least the end effector;
an intention understanding unit that uses the operation information to estimate a target object to be operated by the end effector and a task that is a method of operating the target object intended by the operator ;
an environmental situation determination unit that acquires environmental information of an environment in which the target object exists and the end effector is operated with respect to the target object ;
a motion estimation unit that estimates a motion based on the task, a state of the robot equipped with the end effector, and an object shape of the target object generated based on the environmental information;
an operation amount determination unit that determines an operation amount of the end effector based on the estimated motion and the acquired environmental information ,
The operation amount determination unit
a shape approximation unit that approximates the object shape of the target object to geometric primitives;
a grasp planning unit that generates a grasp plan by comparing the geometric basic elements with registered information of an operation strategy as a strategy for an operation to grasp an object pre-registered for the end effector ,
The grasping planning unit
fitting the object shape of the target object to the geometric primitives to calculate a mapping function between the area of the object shape of the target object and the area of the geometric primitives;
determining a taxonomy capable of performing said task;
a grasping probability map indicating a grasping probability for each operation intention is obtained based on the motion information, the determined taxonomy, and a class ID to which an instance ID assigned to the object shape of the target object belongs; the grasping probability map is mapped to the region of the geometric basic element based on a mapping function of correspondence between the calculated regions; and the grasping plan is generated based on a current wrist posture.
Remote control assistance system. a shape approximation unit that approximates the object shape of the target object to a geometric primitive for each of the instances ID ;
The remote operation assistance system according to claim 1 . The geometric primitive may be one or more .
The remote operation assistance system according to claim 1 . the geometric basic elements include a shape of a region of interest that is an area of the target object that is easy to grasp;
The remote operation assistance system according to claim 1 . The environmental situation determination unit
an RGB sensor for acquiring RGB data;
a depth sensor for acquiring a depth image;
a mask generation unit that uses an instance segmentation technique to generate a mask image based on an instance ID of an object in the image using the color image output by the RGB sensor;
an adder that generates and outputs a depth image for each instance ID using the depth image and the mask image;
a self-position estimation unit that performs image processing on a color image output by the RGB sensor to detect a position and orientation of the RGB sensor , and generates and outputs orientation information based on the position and orientation ;
a three-dimensional reconstruction unit that obtains a point cloud for each instance ID for each class by three-dimensional reconstruction using the orientation information of the RGB sensor output by the self-position estimation unit and the depth image for each instance ID output by the addition unit;
A data integration unit that acquires a point cloud of each instance ID for each key frame, acquires and manages current data at time t, and time series data of past frames t-1, t-2, ..., and integrates these data;
3. The remote operation assistance system according to claim 1, further comprising: a computer of a teleoperation assistance system that teleoperates at least the end effector,
acquiring motion information relating to the motion of an operator who operates at least the end effector ;
Using the motion information , a target object to be operated by the end effector and a task that is a method of operating the target object intended by the operator are estimated;
acquiring environmental information of an environment in which the target object exists and in which the end effector is operated relative to the target object ;
Estimating a motion based on the task, a state of the robot equipped with the end effector, and an object shape of the target object generated by the environmental information;
determining an operation amount of the end effector based on the estimated motion and the acquired environmental information ;
The computer
approximating the object shape of the target object to geometric primitives;
generating a grasp plan by comparing the geometric primitives with registered information of an operation strategy as a strategy for grasping an object pre-registered for the end effector;
The computer
fitting the object shape of the target object to the geometric primitives to calculate a mapping function between the area of the object shape of the target object and the area of the geometric primitives;
determining a taxonomy capable of performing said task;
a grasping probability map indicating a grasping probability for each operation intention is obtained based on the motion information, the determined taxonomy, and a class ID to which an instance ID assigned to the object shape of the target object belongs; the grasping probability map is mapped to the region of the geometric basic element based on a mapping function of correspondence between the calculated regions; and the grasping plan is generated based on a current wrist posture.
Remote control assistance method. A computer of a teleoperation assistance system that teleoperates at least the end effector ,
acquiring motion information relating to at least the motion of an operator operating the end effector;
using the motion information , a target object to be operated by the end effector and a task that is a method of operating the target object intended by the operator are estimated;
acquiring environmental information of an environment in which the target object exists and in which the end effector is operated relative to the target object ;
A motion is estimated based on the task, a state of the robot equipped with the end effector, and an object shape of the target object generated by the environmental information.
determining an operation amount of the end effector based on the estimated motion and the acquired environmental information;
The computer,
approximating the object shape of the target object to geometric primitives;
generating a grasp plan by comparing the geometric primitives with registered information of an operation strategy as a strategy for grasping an object previously registered for the end effector;
The computer,
fitting the object shape of the target object to the geometric primitives to calculate a mapping function between a region of the object shape of the target object and a region of the geometric primitives;
determining a taxonomy capable of performing said task;
a grasping probability map indicating a grasping probability for each operation intention is obtained based on the motion information, the determined taxonomy, and a class ID to which an instance ID assigned to the object shape of the target object belongs; the grasping probability map is mapped to the region of the geometric basic element based on a mapping function of correspondence between the calculated regions; and the grasping plan is generated based on a current wrist posture.
program .