JP7709648B2 - Contact object estimation device, method, and program - Google Patents

Description

One aspect of the present invention relates to a contact object estimation device, method and program for use in a system for remotely controlling a target object using, for example, a robot.

In recent years, various systems have been developed that allow an operator to remotely control an object in a remote location by operating a robot placed in the remote location via a network. This type of system is expected to enable smoother operation by estimating information about the object to be operated in advance.

Therefore, for example, Non-Patent Document 1 or Non-Patent Document 2 propose a technology that acquires the operator's gaze information using a head-mounted display (HMD) and, based on the acquired gaze information, estimates in advance the object that the remote robot is attempting to grasp, thereby reducing delays in the robot's operations.

Negishi et.al “Operation Assistance Using Visual Feedback with Considering Human Intention on Master-Slave Systems” ROBIO 2016 Kojima et.al “Early Estimation of User’s Intention of Tele-Operation Using Object Affordance and Hand Motion in a Dual First-Person Vision” CVPR 2019

However, the technology described in Non-Patent Documents 1 and 2 requires information on the operator's line of sight, which necessitates a separate HMD and a camera for transmitting images. This poses the problem that the system configuration can easily become complicated.

In addition, in the technology described in Non-Patent Documents 1 and 2, the position of an object is measured based on an image captured by a camera installed on the robot, and an object that is likely to be a contact target during operation is estimated. Therefore, if the image contains multiple objects, the positions of all of these objects are measured and the results are combined to estimate the object that is likely to be a contact target, which requires a lot of processing time. As a result, it becomes difficult to maintain real-time control processing.

This invention has been made in light of the above-mentioned circumstances, and aims to provide technology that prevents the system configuration from becoming complicated and improves the real-time processing by enabling the contact object to be estimated in a short period of time even when there are multiple potential contact objects.

In order to solve the above problem, one aspect of the contact object estimation device or estimation method of the present invention is a system that remotely controls an object present in a remote work area by operating a robot at the remote location in response to an operator's operation, and includes an information processing device that operates as the contact object estimation device, the information processing device having a first processing unit or processing step, a second processing unit or processing step, and a third processing unit or processing step.

A first processing unit or process detects a plurality of objects captured in a video frame from video information capturing an image of a work area, and defines a first interference area for each of the detected objects. A second processing unit or process determines whether or not there is interference between the robot and each of the objects based on the first interference area, the operation information of the operator, and a second interference area set for the robot operated by the operation of the operator. A third processing unit or process selects, from among the objects determined to be interfering, the object with which the robot is about to come into contact, based on information representing the positional relationship between the object and the robot.

According to one aspect of the present invention, the object that the robot is about to contact is estimated based on image information of a work area including multiple objects and information on the operation of the robot by the operator. Therefore, for example, it becomes possible to estimate the target to be contacted without using the line of sight information of the operator, which makes it possible to avoid the complication of the system by eliminating the need for additional devices such as an HMD.

In addition, an interference area is defined for each object detected from the video information, and objects that may interfere with the robot are selected based on the interference area of this object and the interference area of the robot arm. Then, from among the selected objects, an object that is most likely to come into contact with the robot arm, for example, is selected. Therefore, even if there are multiple objects in the working area that are potential contact objects, it is possible to quickly estimate the object that is most likely to be the subject of contact, thereby making it possible to maintain real-time processing.

In other words, according to one aspect of the present invention, it is possible to provide a technology that prevents the system configuration from becoming complicated and improves the real-time processing by enabling the object to be contacted to be estimated in a short period of time even when there are multiple potential contact objects.

FIG. 1 is a diagram showing an example of a remote control system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a hardware configuration of a contact object estimation device according to an embodiment of the present invention. FIG. 3 is a block diagram showing an example of a software configuration of the contact object estimation device according to the embodiment of the present invention. FIG. 4 is a flowchart showing an example of the overall processing procedure and processing contents of the operation executed by the control unit of the contact object estimation device shown in FIG. FIG. 5 is a flowchart showing an example of the processing procedure and processing contents related to the detection of object information and the definition of an interference area, among the processing procedures shown in FIG. FIG. 6 is a flowchart showing an example of the procedure and contents of the collision detection process from the procedure shown in FIG. FIG. 7 is a flowchart showing an example of the procedure and contents of the contact target selection process from the procedure shown in FIG. FIG. 8 is a diagram showing an example of a bounding box included in video information. FIG. 9 is a diagram illustrating an example of hand information stored in the hand information storage unit illustrated in FIG. 3 . FIG. 10 is a diagram showing an example of object information stored in the object information storage unit shown in FIG. 3. As shown in FIG.

Below, an embodiment of the present invention is described with reference to the drawings.

[One embodiment]
(Configuration example)
(1) System FIG. 1 is a diagram showing an example of a remote control system according to an embodiment of the present invention.

In one embodiment of the remote control system, a robot arm RB placed at a remote work site or the like is remotely controlled by an operator US via a network NW to perform a desired operation on an object TL to be operated, such as a part, tool, or device.

In addition to the robot arm RB and the robot control device RU that controls its operation, a camera CM and a contact object estimation device EU are placed at the remote location. Of these, the camera CM is, for example, an RGB-D camera, and captures an area including the objects TL at the site and the robot arm RB, and outputs video information of the area.

On the other hand, the operating side is provided with an operating side terminal device OU and a monitor MT, and furthermore, the operator US is equipped with a motion sensor SS for detecting, for example, hand movements. The monitor MT is used to display video information captured by the camera CM at a remote location.

The motion sensor SS is, for example, an infrared sensor, and detects the movement of the operator US's hand as a change in the position of a specific part of the hand. For example, the motion sensor SS detects the position of the center of the operator US's hand, the position of each finger joint, the position of each fingertip, and the position of the wrist, each as three-dimensional coordinate information. The motion sensor SS also detects whether the operator US is using his/her left or right hand, and whether he/she is using the front or back of the hand to operate. The front of the hand refers to the back of the hand, and the back of the hand refers to the palm side.

The operating side terminal device OU has a function of receiving video information transmitted from a remote contact object estimation device EU via a network NW and displaying it on the monitor MT, and a function of transmitting sensor information representing the hand movement of the operator US detected by the motion sensor SS to the remote contact object estimation device EU via the network NW. The sensor information includes three-dimensional coordinate information indicating the change in position of each part of the hand of the operator US, as well as information indicating whether the hand is left or right and information indicating the front or back orientation of the hand.

(2) Contact object estimation device EU
2 and 3 are block diagrams showing an example of a hardware configuration and a software configuration, respectively, of the contact object estimation device EU.

The contact object estimation device EU is composed of an information processing device such as a personal computer or a server computer, and has a control unit 1 using hardware processors such as a central processing unit (CPU) and a graphics processing unit (GPU). A storage unit having a program storage unit 2 and a data storage unit 3, a camera interface (hereinafter the interface is abbreviated as I/F) unit 4, a communication I/F unit 5, and a robot I/F unit 6 are connected to the control unit 1 via a bus 7. The control unit 1 may be configured using a programmable logic device (PLD), a field programmable gate array (FPGA), or the like.

The camera I/F unit 4 is used to capture video information output from the camera CM. The communication I/F unit 5 transmits video information captured by the camera CM to the operating terminal device OU via the network NW in accordance with a communication protocol defined by the network NW, and receives sensor information transmitted from the operating terminal device OU via the network NW. The robot I/F unit 6 outputs information representing the selection result of the contact target to the robot control device RU.

The program storage unit 2 is configured by combining a non-volatile memory such as a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for executing various control processes according to one embodiment, in addition to middleware such as an operating system (OS). Hereinafter, the OS and each application program will be collectively referred to as a program.

The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD, which can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium, and includes an image information storage unit 31, a hand information storage unit 32, and an object information storage unit 33 as the main storage areas required to implement one embodiment.

The video information storage unit 31 temporarily stores the video information obtained from the camera CM for the object detection processing described below.

The hand information storage unit 32 is used to store sensor information representing the hand movement of the operator US detected by the motion sensor SS and a velocity vector of the hand movement generated based on this sensor information. The hand information storage unit 32 also stores information representing the interference area of the operator hand of the robot arm RB.

The object information storage unit 33 is used to store information representing multiple objects detected from the video information of the camera CM in chronological order. The information representing the objects includes an object ID uniquely assigned to the detected object, its position and speed, information representing the object's interference area, a speed vector, and information representing the interference determination result. Details of the object information will be described in the operation example.

The control unit 1 includes, as processing functions necessary for implementing one embodiment, a video information acquisition processing unit 11, a video information transmission processing unit 12, a hand information generation processing unit 13, an object detection processing unit 14, an interference area definition processing unit 15, an interference determination processing unit 16, a contact target selection processing unit 17, and an estimated information output processing unit 18. All of the above processing units 11 to 18 are realized by causing the hardware processor of the control unit 1 to execute application programs stored in the program storage unit 2.

The video information acquisition processing unit 11 acquires the time-series video information output from the camera CM via the input/output I/F unit 4, and performs a process of temporarily storing each of the acquired video information in the video information storage unit 31.

Video information consists of video frames including bounding boxes, and point cloud information. A bounding box displays the position of an object in a video frame as a rectangular frame, and is represented by the coordinates of the upper left corner and the lower right corner of the rectangular frame, for example, when the coordinates of the upper left corner of the video frame are (0, 0). Point cloud information is, for example, depth information, and represents the distance distribution of each pixel from the camera CM of the video frame.

The video information transmission processing unit 12 performs processing to transmit video frames including bounding boxes from the video information captured by the camera CM from the communication I/F unit 5 to the operating side terminal device OU.

The hand information generation processing unit 13 receives sensor information transmitted from the operating side terminal device OU via the communication I/F unit 5, and calculates a velocity vector of the hand movement based on the received sensor information. The hand information generation processing unit 13 then stores in the hand information storage unit 32 the information included in the received sensor information indicating the positions of each part of the hand, information indicating whether the hand is the left or right hand, and information indicating whether the operating side is the front or back of the hand, as well as the calculated velocity vector.

The hand information generation processing unit 13 also performs processing to store information representing a pre-set interference area based on the center of the operating hand of the robot arm RB in the hand information storage unit 32. A specific method for setting the information representing the interference area will be described in the operation example.

The object detection processing unit 14 reads the video information from the video information storage unit 31 frame by frame, and detects the multiple objects and information representing the operating hand of the robot arm RB that appear in the read video frames. This detection is performed using a bounding box. The object detection processing unit 14 outputs the video data and point cloud information of the area surrounded by the detected bounding box.

The interference area definition processing unit 15 performs a process of excluding objects that are outside the operational range of the robot arm RB for each object detected by the object detection processing unit 14. In addition, the interference area definition processing unit 15 performs a process of defining an interference area for each object that was not excluded by the above exclusion process. An example of the above exclusion process and the process of defining an interference area will be described in the operation example.

The interference determination processing unit 16 determines whether or not there is interference between each object and the operating hand of the robot arm RB corresponding to the hand information of the operator US, using a determination formula prepared in advance for each object, based on the hand information of the operator US stored in the hand information storage unit 32 and the object information of multiple objects stored in the object information storage unit 33. Then, interference determination information representing the determination result is added to the object information storage unit 33. An example of the interference determination process will also be described in the operation example.

The contact target selection processing unit 17 first reads the hand information of the operator US stored in the hand information storage unit 32 and the object information on the object determined to be in interference by the interference determination processing unit 16. Next, based on the above-mentioned information read, the contact target selection processing unit 17 obtains, for each object determined to be in interference, information indicating which of the front (back side of the hand) or the back (palm side) of the operator US's hand faces the object, and a center capture degree, which is an index indicating how well the hand of the operator US captures the center of the object. Then, the contact target selection processing unit 17 performs a score calculation based on the information indicating which of the front and back of the hand faces and the center capture degree, and performs a process of selecting an object that is most likely to be contacted by the operator hand of the robot arm RB based on the calculated cost.

The estimated information output processing unit 18 performs processing to output object information regarding the object selected by the contact target selection processing unit 17 from the robot I/F unit 6 to the robot control device RU as auxiliary information for the operation of the robot arm RB.

(Example of operation)
Next, an example of the operation of the contact object estimation device EU configured as above will be described.
FIG. 4 is a flowchart showing an example of the overall processing procedure and processing contents executed by the control unit 1 of the contact object estimation device EU.

(1) Acquisition and transmission of video information The control unit 1 of the contact object estimation device EU monitors the start of remote operation in step S10. In this state, assume that the operator US starts the system to remotely operate an object using the robot arm RB. Then, under the control of the video information acquisition processing unit 11, the control unit 1 of the contact object estimation device EU acquires video information of the work area captured by the camera CM via the camera I/F unit 4 in step S11, and stores the acquired video information in the video information storage unit 31.

At this time, the imaging range of the camera CM is set to include multiple objects present in the work area and the operator's hand (gripping part) of the robot arm RB. Therefore, the video information acquisition processing unit 11 obtains video information including images of multiple objects present in the work area and an image of the operator's hand of the robot arm RB in a video frame. In addition, since an RGB-D camera is used as the camera CM, video information including video data with bounding boxes and point cloud information is obtained.

When the video information is acquired, the control unit 1 of the contact target estimation device EU, under the control of the video information transmission processing unit 12, reads out only the video data from the video data and point cloud information contained in the video information stored in the video information storage unit 31 in step S12. Then, the read out video data is transmitted from the communication I/F unit 5 to the operating side terminal device OU via the network NW.

As a result, on the operating side, the image data is displayed on the monitor MT by the operating side terminal device OU. Therefore, the operator US can then remotely control the object while watching the image of the work area displayed on the monitor MT.

(2) Acquisition of Sensor Information, and Generation and Storage of Hand Information When the operator US moves his/her hand to remotely operate the robot arm RB, the control unit 1 of the contact object estimation device EU, under the control of the hand information generation processing unit 13, first receives sensor information representing the hand movement of the operator US detected by the movement sensor SS at a predetermined cycle via the communication I/F unit 5 in step S13. The sensor information includes, for example, three-dimensional coordinate information representing the position of the center of the hand of the operator US, the positions of the joints of each finger, the positions of each fingertip, and the position of the wrist, as well as information indicating whether it is the left or right hand.

Next, in step S14, each time sensor information is received, the hand information generation processing unit 13 calculates a velocity vector from the difference between the three-dimensional coordinate information representing the position of each part of the hand of the operator US contained in the sensor information and the three-dimensional coordinate information representing the position of each part of the hand contained in the sensor information received at the previous timing.

The hand information generation processing unit 13 then stores in the hand information storage unit 32 the three-dimensional coordinate information representing the position of each part included in the received sensor information, information indicating whether the hand is the left or right hand, and the calculated velocity vector. The hand information generation processing unit 13 also stores in the hand information storage unit 32 information representing an interference area set for the operating hand of the robot arm RB. The information representing the interference area is set, for example, as a sphere representing a range of 90 mm in diameter from the center of the palm of the operating hand of the robot arm RB. Figure 9 shows an example of hand information stored in the hand information storage unit 32.

(3) Detection of Objects and Definition of Interference Areas When the above-mentioned video information is acquired, in step S15, the control unit 1 of the contact target estimation device EU detects objects from the video data for each video frame and executes a process of defining an interference area for each detected object as follows.

Figure 5 is a flowchart showing an example of the processing steps and processing contents of a series of processes performed by the object detection processing unit 14 and the interference area definition processing unit 15 in step S15 above.

In step S20, the object detection processing unit 14 reads a video frame from the video information storage unit 31 and detects multiple objects and the operating hand of the robot arm RB that appear in the read video frame based on the bounding boxes assigned to them. Then, in step S21, the object detection processing unit 14 assigns a unique object ID to each of the detected objects, and outputs the video data and point cloud information of the area surrounded by the bounding box for each object.

Figure 8 shows an example of the detected objects TL1, TL2, TL3 and the bounding boxes B1, B2, B3 representing the image areas of the objects TL1, TL2, TL3. As shown in the figure, the shapes and sizes of the bounding boxes B1, B2, B3 change depending on the shapes and sizes of the objects TL1, TL2, TL3.

First, in step S22, the interference area definition processing unit 15 selects one of the detected objects and calculates the distance between this object and the operator's hand of the robot arm RB. This distance is calculated based on the three-dimensional coordinate information of the object and the operator's hand of the robot arm RB. The three-dimensional coordinate information of the object and the operator's hand of the robot arm RB is obtained from the coordinate information and point cloud information indicating the position of each bounding box.

In step S23, the interference area definition processing unit 15 then compares the calculated distance with a preset threshold value. If the distance exceeds the threshold value, the interference area definition processing unit 15 determines that the object cannot be operated by the robot arm RB, and excludes the object from the objects to be determined.

In step S24, the interference area definition processing unit 15 then performs a process of defining an interference area for each object that was not excluded by the above exclusion process, that is, for each object whose distance is within the threshold range. The interference area is defined, for example, by using a sphere based on the center position of the object. To calculate the diameter of the sphere that represents the interference area, first, both ends of the object that exists within the bounding box are searched for. Next, from among the candidates for the points that are the both ends of the object, a straight line passing through the center position of the bounding box is drawn to connect the two points with the greatest distance, and the distance between the two selected points is set as the diameter of the sphere.

Finally, in step S25, the interference area definition processing unit 15 stores the information representing the object obtained by the above series of processes in the object information storage unit 33. The information representing the object stored at this time includes the object ID, the position and speed of the object, the sphere information representing the interference area of the object, and the speed vector. Figure 10 shows an example of the object information stored in the object information storage unit 33.

(4) Determination of Whether or Not Interference Will Occur Next, under the control of the interference determination processing unit 16, in step S16, the control unit 1 of the contact object estimation device EU executes a process of determining whether or not there is interference between each detected object and the operating hand of the robot arm RB as follows.

Figure 6 is a flowchart showing an example of the processing procedure and processing content of the determination processing performed by the interference determination processing unit 16 in step S16 above.

First, in step S30, the interference determination processing unit 16 reads a list of object information from the object information storage unit 33, and also reads hand information representing the hand operation movement of the operator US from the hand information storage unit 32. Then, in steps S31 to S36, the interference determination processing unit 16 executes a process of determining the presence or absence of interference based on the read object information and hand information using a determination formula prepared in advance for each object.

That is, the elements used in collision detection are the position vectors and velocity vectors of the hand of the operator US and the object. The position vectors and velocity vectors of the object A and the hand H at a certain time t [seconds] are respectively
This is expressed as:

Similarly, the relative position and the relative velocity between the hand of the operator US and the object are expressed as follows:
This is expressed as:

When the current time is t ₀ seconds, the relative position between the hand of the operator US and the object at any time t [seconds] is
It can be expressed as:

In this formula, in order to handle three-dimensional movement, the position vector and velocity vector are expressed by three axes 1, 2, and 3, respectively, as follows:
It will look like this.

Solving equation (1), we get
…(2)
It becomes.

Here, a, b, and c are
It can be expressed as follows.

The solution, interference start time _t1 and interference end time _t2 , are respectively
It becomes.

In the above formula
The value of indicates the sum of the radius of the sphere representing the interference region included in the object information and the radius of the sphere representing the interference region of the hand of the operator US included in the hand information.

The interference determination processing unit 16 determines that "interference exists" if the value in the root of the above formula (2) is positive, and determines that "no interference exists" if the value is negative. When it is determined that "interference exists," the above determination formula is calculated on the assumption that there is uniform linear motion between the interference start time _t1 [seconds] and the interference end time _t2 [seconds]. As a result, the interference position can be determined from the current position, speed, and interference time.
It can be calculated as:

The interference determination processing unit 16 additionally stores, in the object information storage unit 33, interference information including information indicating the result of the determination of whether or not there is interference for the object being determined, and the interference time and interference position obtained by the above calculation, in association with the object ID.

The collision detection processing unit 16 then performs the above series of collision detection processes for each of the objects included in the object information list.

(5) Selection of object to be contacted Finally, in step S17, under the control of the contact target selection processing unit 17, the control unit 1 of the contact target estimation device EU executes a process of selecting an object that is most likely to be a contact target of the operating hand of the robot arm RB as follows.

Figure 7 is a flowchart showing an example of the processing procedure and processing content of the selection processing performed by the contact target selection processing unit 17.

That is, first, in step S41, the contact target selection processing unit 17 reads hand information of the operator US from the hand information storage unit 32, and also reads object information related to the objects determined to be interfering from the object information storage unit 33. Then, the contact target selection processing unit 17 performs score calculation for each of the objects determined to be interfering in steps S42 to S43.

For example, for each object determined to have interfered, the contact target selection processing unit 17 calculates information FB indicating whether the front or back of the hand of the operator US faces the object, and a central capture degree CC, which is an index indicating how well the hand of the operator US captures the center of the object.

Next, in step S44, the contact target selection processing unit 17 calculates a cost Cost indicating the possibility of contact based on the information FB indicating whether the hand is facing the front or back and the center capture degree CC.
Cost＝CC×FB
It is calculated as follows.

Here, the central capture degree CC is the distance when the interference areas of the hand of the operator US and the object are closest to each other,
It is calculated as:

That is, the central capture degree CC is calculated as the distance between the hand of the operator US and the object at (t ₁ + t ₂ )/2, which is the midpoint between the start time t ₁ and the end time t ₂ of the period in which interference occurs.

Information FB, which indicates whether the front or back of the hand of the operator US is facing the object, is a parameter that is preset so that the value when the palm of the hand is facing the object is smaller than the value when the back of the hand is facing the object; for example, "1.0" is set for the back of the hand and "1.2" is set for the front of the hand. This is because contact and grasping of an object is generally performed with the palm of the operator US, that is, the palm side of the robot arm RB. The values of all of the above parameters can be adjusted arbitrarily.

Then, the contact target selection processing unit 17 compares the above-mentioned costs between the objects, and determines that the object with the smallest cost value is the object most likely to be grasped by the hand of the operator US, and selects it as the object to be contacted.
In addition, when making the selection, the one object with the highest possibility may be selected as described above, but a predetermined number of objects with the highest possibility may also be selected.

In step S18, the control unit 1 of the contact object estimation device EU outputs object information relating to the selected object from the robot I/F unit 6 to the robot control device RU under the control of the estimated information output processing unit 18. The robot control device RU controls the operation of the robot arm RB taking into account the object information.

(Action and Effects)
As described above, in one embodiment, the contact target estimation device EU detects each object shown in the image frame from the image information of the working area captured by the camera CM together with the hand part of the robot arm RB, and after excluding objects that are at a distance that cannot be operated by the robot arm RB from each of the detected objects, defines an interference area for each object that has not been excluded. Meanwhile, in parallel with this, hand information indicating the position of the hand of the operator US is generated from sensor information obtained by the motion sensor SS attached to the operator US.

Next, for each of the above objects, it is determined whether the interference area of the object interferes with the interference area of the operator's hand of the robot arm RB, based on the position vector and velocity vector of the object and the hand of the operator US. Then, for each object determined to interfere, a cost Cost is calculated based on information FB indicating whether the front or back of the operator's hand is facing, and a center capture degree CC indicating how well the operator's hand captures the center of the object, and the object that is most likely to be grasped by the hand of the robot arm RB is selected from among the above objects based on the calculated cost Cost, and the object information related to the selected object is used for robot control.

Therefore, according to one embodiment, it becomes possible to estimate the object that the robot arm RB is trying to grasp in response to the operation of the operator US without using the line of sight information of the operator US. As a result, it becomes possible to eliminate the need for additional devices such as an HMD and to avoid complicating the system.

In addition, an interference area is defined for each object detected in the video, objects that may interfere with the interference area set in the hand of the robot arm RB are identified, and from among the identified objects, the object that is most likely to be grasped by the robot arm RB is selected. As a result, even if there are multiple objects in the working area that are candidates for operation targets, it is possible to quickly estimate the object to be operated, thereby maintaining real-time processing.

Furthermore, when defining the interference area for each of the above objects, objects whose distance between each object and the robot arm RB exceeds a threshold value are excluded in advance, and the interference area is defined for only the objects that have not been excluded. This makes it possible to reduce the time required for the process of defining the interference area, thereby further improving the real-time nature of the process.

[Other embodiments]
(1) In the embodiment, the contact object estimation device EU is independently provided at a remote location, but all or part of the functions of the contact object estimation device EU may be provided in the robot control device RU. The contact object estimation device EU does not necessarily have to be provided at the remote location, and may be provided at the operating side, or may be provided on the Web or on the cloud. In this case, the processing functions of the contact object estimation device EU may be distributed to information processing devices provided at the remote location and the operating side, or a server device provided on the Web or on the cloud.

(2) In the embodiment, an example has been described in which an RGB-D camera is used. However, a normal camera may also be used. In this case, by providing the contact object estimation device EU with a function for detecting an object image from video information captured by the camera and a function for generating depth information from the detected object image, it becomes possible to estimate an object to be contacted in the same manner as in the embodiment.

(3) As a means for detecting an object image, for example, a machine learning model that takes a video frame as input and outputs information representing the class and position of an object appearing in the video frame can be used. Also, as a means for generating depth information, for example, a machine learning model for detecting distance information that takes a video frame as input and outputs depth information representing the distance distribution of each pixel from the camera CM of the image appearing in the video frame can be used.

(4) In one embodiment, the case where objects such as tools, parts, and devices are detected as contact targets is described as an example. However, the contact target may also be other living things such as people, animals, and plants.

(5) The number of cameras CM may not be limited to one, but may be multiple, and the images from these cameras CM may be selectively used or combined to detect an object. Furthermore, an acceleration sensor may be used as a motion sensor to detect the motion of the operator, and the body parts to be detected for motion detection may be not only the hands, but also the legs, waist, and neck.

In addition, the functional configuration, processing procedures, processing contents, etc. of the contact object estimation device can be modified in various ways without departing from the spirit and scope of this invention.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

In short, this invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriate combinations of multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all of the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

EU...contact object estimation device CM...camera RU...robot control device RB...robot arm US...operator OU...operating side terminal device SS...motion sensor MT...monitor 1...control unit 2...program storage unit 3...data storage unit 4...camera I/F unit 5...communication I/F unit 6...robot I/F unit 7...bus 11...video information acquisition processing unit 12...video information transmission processing unit 13...hand information generation processing unit 14...object detection processing unit 15...interference area definition processing unit 16...interference determination processing unit 17...contact object selection processing unit 18...estimated information output processing unit 31...video information storage unit 32...hand information storage unit 33...object information storage unit

Claims (8)

1. A contact object estimation device used in a system for remotely controlling an object present in a work area at a remote location by operating a robot at the remote location in response to an operation by an operator, comprising:
a first processing unit that detects a plurality of the objects appearing in a video frame of the video information captured in the work area, and defines a first interference region for each of the detected plurality of the objects;
a second processing unit that determines whether or not there is interference between each of the plurality of objects and the robot, based on the first interference region defined for the object, a second interference region set for the robot, and operation information of the operator; and
and a third processing unit that selects the object that the robot is about to come into contact with from among the plurality of objects determined to be interfering, based on information representing a positional relationship between the object and the robot. The first processing unit includes:
a processing unit that excludes an object from the plurality of objects, the object being located beyond a preset range of distance between the object and the robot;
The contact object estimation device according to claim 1 , further comprising: a processing unit that defines the first interference region for the object that has not been excluded. The contact object estimation device of claim 1, wherein the second processing unit calculates a relative position between the object and the robot at any time based on the position vectors and velocity vectors of the object and the robot, and determines whether or not there is interference between the object and the robot based on the calculated relative position. The contact object estimation device of claim 3, wherein the second processing unit calculates the relative position in three-dimensional space using the position vector of three axes and the velocity vector of three axes. The contact target estimation device of claim 1, wherein the third processing unit selects the object that the robot is about to contact based on the distance between the robot and the object when the robot and the object are closest to each other. The contact target estimation device of claim 5, wherein the third processing unit selects the object that the robot is about to contact by taking into account the distance between the robot and the object when the robot and the object are closest to each other, and information indicating whether the object is facing the front or back of the operator's hand. 1. A contact target estimation method executed by an information processing device used in a system for remotely controlling an object present in a work area at a remote location by operating a robot at the remote location in response to an operation by an operator, comprising:
a first processing step of detecting a plurality of objects captured in a video frame of the video information captured in the work area, and defining a first interference region for each of the detected plurality of objects;
a second processing step of determining whether or not there is interference between each of the plurality of objects and the robot, based on the first interference region defined for the object, a second interference region set for the robot, and operation information of the operator;
and a third processing step of selecting the object that the robot is about to come into contact with from among the plurality of objects determined to be interfering, based on information representing a positional relationship between the object and the robot. A program for causing a processor provided in a contact target estimation device to execute at least one of the first processing unit, the second processing unit, and the third processing unit provided in the contact target estimation device described in any one of claims 1 to 6.