JP7761340B2 - Control device, control method, program, and robot control system - Google Patents

Description

The present invention relates to a control device, a control method, a program, and a robot control system.

In recent years, there are high expectations for robots to be used in a variety of fields. One method of controlling a robot is for a person to operate it from a remote location. One operation method is to expect the robot arm to move exactly as a person (operator) operates (moves) it from a remote location using their arm or hand. This type of remote operation of a robot arm can be used, for example, to allow robots to work in place of humans in disaster areas or places that are difficult for humans to enter (including places that are dangerous for humans or that may have adverse effects on the human body). It is also expected to be used in "remote surgery," in which surgeons perform surgery from a remote location without traveling to the site. It is also expected to be used for experiential operation in "online tours" conducted via communication lines without people having to go to the site in person.

"Teleoperation," in which a robot arm in a remote location moves in accordance with the operations (movements) of a person (operator), allows for intuitive operation by the person (operator), with no (or very few) special skills to be acquired.

Patent Document 1 describes a robot control method. In particular, paragraph 0034 of Patent Document 1 states, "The operation computer 10 generates commands for the robot 3 mainly based on the motion of the operator 40." Furthermore, paragraph 0266 of Patent Document 1 states, "It operates without delay in the virtual space, simulates the movement of the robot 3 using a physical simulator, and uses the simulation results to synchronize and operate the operator 40 and avatar 41 in the virtual space. Data indicating the movement of the avatar 41 is then stored in memory and sequentially transmitted to the robot 3."

Patent Document 2 describes a technology for guiding a robot arm. In particular, paragraph 0192 of Patent Document 2 states, "The robot arm 6 is guided by a motion controller GEN_TRAJ that takes into account various constraints for controlling the robot arm. In particular, this controller takes into account the planning of a set trajectory and the user's definition of the zone to be treated." Furthermore, paragraph 0187 of Patent Document 2 states, "An advantage of the method of the present invention is that it is possible to generate a new modeled surface in less than 30 ms. These response times make it possible, in particular, to control the movement of the robot arm 6 at a speed sufficient to anticipate collisions or sudden movements in order to leave the area as quickly as possible."

Patent Document 3 describes a technology for vision-based teleoperation of a robot system. In particular, paragraph 15 of Patent Document 3 states, "FIG. 3 illustrates an example of a system for tracking a human hand in real time and controlling a robot to perform corresponding actions, according to at least one embodiment." Furthermore, paragraph 20 of Patent Document 3 states, "FIG. 4 illustrates an example of a color glove used to acquire hand pose and segmentation. In at least one embodiment, glove 402 includes five fingers 404, 406, 408, 410, and 412, each colored with a different colored cloth. In at least one embodiment, unique colors are printed to minimize annotation generation using OpenCV color thresholding. In at least one embodiment, the color of the back of the hand 414 uniquely determines the hand pose." Furthermore, paragraph 23 of Patent Document 3 states, "In at least one embodiment, hand pose can be estimated by three unique keypoints represented by three different colored blobs on the back of the glove."

Patent Document 4 describes a technique for tracking hands and fingers in a surgical system. For example, Figures 2A to 2D of Patent Document 4 disclose an example of attaching sensors to the surgeon's fingers. Furthermore, paragraph 0052 of Patent Document 4 states, "The master finger tracking grip 270 includes finger-worn sensors 211, 212 (sometimes referred to as finger- and thumb-worn sensors 211, 212) to independently track the location (position and orientation in one embodiment) of the tip of the index finger 292B and the tip of the thumb 292A, i.e., track the locations of the two fingers of the surgeon's hand. Thus, in contrast to tracking the location of a master tool grip in known minimally invasive surgical systems, the location of the hand itself is tracked." Furthermore, paragraph 0056 of Patent Document 4 states, "The locations (positions and orientations) of thumb 292A and index finger 292B in FIGS. 2A to 2D are mapped to grip closure parameters, e.g., standardized grip closure values, which are used to control the grip of a teleoperated slave surgical instrument coupled to master finger-tracking grip 270. In particular, the sensed locations of thumb 292A and index finger 292B are mapped to grip closure parameters by hand tracking controller 130." Furthermore, paragraph 0057 of Patent Document 4 states, "Accordingly, the location of a portion of the hand of the surgeon 181 is tracked. Based on the tracked location, a system control parameter for the minimally invasive surgical system 100, i.e., a grip closure parameter, is generated by the hand tracking controller 130 and provided to the system controller 140. The system controller 140 uses the grip closure parameter when generating a system command that is sent to the teleoperated slave surgical instrument. The system command instructs the teleoperated surgical instrument to configure its end effector to have a grip closure that corresponds to the grip closure parameter. Thus, the minimally invasive surgical system 100 uses the grip closure parameter to control the operation of the teleoperated slave surgical instrument of the minimally invasive surgical system 100."

International Publication No. 2018/097223 Special Publication No. 2022-528499 Japanese Patent Application Laid-Open No. 2021-049635 Special Publication No. 2013-510673

Remote control, which involves moving a robotic arm in a remote location in accordance with the instructions of a human (operator) using background related technologies, is used in fields such as surgery and cell culture, but requires specialized equipment for each application, which is expensive.

There have been attempts to achieve remote control using a dedicated controller and a general-purpose robot arm, but problems exist with the robot arm's low control precision and poor operability. Furthermore, using a dedicated controller requires getting used to, making it difficult for the person (operator) to operate it intuitively.

In the related background technology, sensors are attached to two fingers of the surgeon to independently track the positions of the tips of the thumb and index finger. However, this related technology cannot operate a robotic arm based on the movement of the entire human (operator) hand.

The present invention was made in consideration of the above circumstances, and the problem to be solved is to control a robot arm with high precision. Furthermore, the problem to be solved by the present invention is to control a robot arm in response to the delicate movements of a person's (operator's) entire hand (including finger joints, etc.).

The present invention aims to provide a control device, control method, program, and robot control system that solve the above-mentioned problems.

[1] In order to solve the above problem, one aspect of the present invention provides a control device comprising: an input device-side interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand; a control function unit that generates and outputs a control signal for controlling the movement of a robot arm corresponding to the hand based on the hand position, posture, and bending angles of the joints; and an output device-side interface unit that transmits the control signal generated by the control function unit to the robot arm and receives status information on the robot arm at a predetermined timing and passes it to the control function unit; the status information on the robot arm is information representing the hand position, posture, and bending angles of the joints of each finger on the robot arm, which correspond to the hand position, posture, and bending angles of the joints; and the control function unit generates the control signal at the predetermined timing based on the status information passed from the output device-side interface unit so that the status of the user's hand and the status information on the robot arm are consistent.

[2] Another aspect of the present invention is a control device according to [1] above, wherein the control function unit includes a noise correction function unit that, when noise is detected based on the temporal progression of at least one of the information on the position of the user's hand, the information on the posture of the user's hand, and the information on the bending angles of the joints of each finger of the user's hand, corrects any of the information on the position of the user's hand, the information on the posture of the user's hand, and the information on the bending angles of the joints of each finger of the user's hand that caused the noise so as to eliminate or reduce the detected noise.

[3] Another aspect of the present invention is that in the control device of [1] or [2] above, the input device side interface unit acquires signals representing information on the bending angles of all joints in the user's hand, and the control function unit generates and outputs the control signal based on the information on the bending angles of all joints.

[4] In one aspect of the present invention, the control device according to any one of [1] to [3] above further includes a capturing function unit that stores the time series of the control signals generated by the control function unit, and, when requested, reads the stored time series of the control signals and controls the output device-side interface unit to transmit the time series of the control signals to the robot arm.

[5] Another aspect of the present invention is a control method including the steps of: an input device-side interface unit acquiring signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand; a control function unit generating and outputting a control signal for controlling the movement of a robot arm corresponding to the hand based on the hand position, posture, and bending angles of the joints; and an output device-side interface unit transmitting the control signal generated by the control function unit to the robot arm and receiving status information on the robot arm at a predetermined timing and passing it to the control function unit, wherein the status information on the robot arm is information representing the hand position, posture, and bending angles of each finger joint on the robot arm side corresponding to the hand position, posture, and bending angles of the joints, and the control function unit generating the control signal at the predetermined timing based on the status information passed from the output device-side interface unit so that the status of the user's hand and the status information on the robot arm side are consistent.

[6] Another aspect of the present invention is a program for causing a computer to function as a control device, comprising: an input device-side interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand; a control function unit that generates and outputs control signals for controlling the movement of a robot arm corresponding to the hand based on the hand position, posture, and bending angles of the joints; and an output device-side interface unit that transmits the control signals generated by the control function unit to the robot arm and receives status information on the robot arm at a predetermined timing and passes it to the control function unit, wherein the status information on the robot arm is information representing the hand position, posture, and bending angles of each finger joint on the robot arm that correspond to the hand position, posture, and bending angles of the joints, and the control function unit generates the control signals at the predetermined timing based on the status information passed from the output device-side interface unit so that the status of the user's hand and the status information on the robot arm are consistent.

[7] Another aspect of the present invention is a robot control system comprising: a control device according to any one of [1] to [4] above; a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device interface unit of the control device; and a glove that transmits signals representing information on the bending angles of the joints of each finger on the user's hand to the input device interface unit of the control device.

[8] Another aspect of the present invention is a robot control system comprising: a control device according to any one of [1] to [4] above; and a robot arm that operates based on the control signal transmitted from the output device-side interface unit of the control device and transmits the status information of its own device to the output device-side interface unit at the predetermined timing.

[9] Another aspect of the present invention is a robot control system comprising: a control device according to any one of [1] to [4] above; a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device interface unit of the control device; a glove that transmits signals representing information on the bending angle of each finger joint in the user's hand to the input device interface unit of the control device; and a robot arm that operates based on the control signal transmitted from the output device interface unit of the control device and transmits the status information of its own device to the output device interface unit at the predetermined timing.

According to the present invention, the control function unit can generate control signals based on status information passed from the robot arm so that the status of the user's hand matches the status information on the robot arm. In other words, the control device can control the robot arm with high precision.

1 is a block diagram showing a schematic functional configuration of a robot control system according to a first embodiment of the present invention. FIG. 2 is a schematic diagram showing an example of the appearance of a glove used in the robot control system according to the first embodiment. FIG. 2 is a schematic diagram showing an example of the appearance of a tracker used in the robot control system according to the first embodiment. FIG. 1 is a schematic diagram showing an example of the appearance of a robot arm used in a robot control system according to a first embodiment. FIG. 3 is a block diagram showing a more detailed functional configuration of a control function unit according to the first embodiment. 3 is a schematic diagram for explaining the principle when the control function unit according to the first embodiment controls the robot arm 7. FIG. 10 is a graph showing the transition of the speed on the input side before the noise correction function unit according to the first embodiment corrects noise. 10 is a graph showing the transition of the speed on the input side after the noise correction function unit according to the first embodiment corrects noise. 10 is a schematic diagram showing the operation of capturing and reproducing motion by the motion capture teaching function unit according to the first embodiment. FIG. 3 is a flowchart showing a procedure of processing (control) of the control device according to the first embodiment. FIG. 10 is a functional block diagram showing an example of a minimum configuration of a control device according to a second embodiment. FIG. 10 is a functional block diagram showing the internal functional configuration of a control function unit in a second embodiment. FIG. 2 is a block diagram showing the functional configuration of a computer for realizing the first or second embodiment.

Next, an embodiment of the present invention will be described with reference to the drawings. In the embodiment described below, the control device controls the robot arm to move in accordance with the hand movement of a human (operator) detected by the input device. Specifically, the control device acquires the status of the input device and generates speed information for each part of the output device (robot arm) based on that information. This speed information is control information. The control device controls the output device by transmitting the generated control information to the output device. In this embodiment, the output device detects its own status and feeds back that status information to the control device. The control device can improve control accuracy by generating control information based on the fed-back status information. However, because frequent feedback of status information from the output device to the control device increases the load on the overall control, feedback of status information is performed only at specified times.

[First embodiment]
1 is a block diagram showing a schematic functional configuration of a robot control system according to this embodiment. As shown in the figure, the robot control system 1 includes a control device 2, a glove 3, a tracker 4, a robot arm 7, and a communication line 9. Of these components, the glove 3 and the tracker 4 are also called "input devices," and are devices that input (supply) signals that serve as the basis for controlling the robot arm 7 to the control device 2. The robot arm 7 is also called an "output device," and is a device that is controlled by control signals output by the control device 2.

Note that at least one of the glove 3 and the tracker 4 may be located outside the robot control system 1. In this case, the glove 3 and the tracker 4 each input (supply) a signal (the content of the signal will be explained later) to the control device 2. The robot arm 7 may also be located outside the robot control system 1. In this case, the movement of the robot arm 7 is controlled by a control signal output from the control device 2. The robot control system 1 may also be configured not to include the communication line 9. In this case, signals are exchanged between the control device 2 and the robot arm 7 without going through the communication line 9.

The general functional configuration of the control device 2 is as follows. As shown in the figure, the control device 2 includes an input device interface unit 21, a control function unit 22, and an output device interface unit 23.

At least some of the functions of the control device 2, glove 3, tracker 4, and robot arm 7 may be implemented using electronic circuits. At least some of the functions of the glove 3, tracker 4, and robot arm 7 may also be implemented using a computer and a program. The control device 2, glove 3, tracker 4, and robot arm 7 may include storage means for at least temporarily storing information. The storage means may include non-volatile storage means such as a magnetic hard disk drive (HDD) or a solid state drive (SSD). The storage means may also include volatile memory.

The functions that make up the robot control system 1 are described below.

The control device 2 controls the movement of the robot arm 7 based on signals received from the glove 3 and the tracker 4. In addition, if the robot arm 7 has a robot hand, the control device 2 also controls the movement of the robot hand held by the robot arm.

The control device 2 may control one robot arm 7 in response to the movement of only one hand of the human (operator). In this case, the control device 2 receives signals from one glove 3 that represent the bending angles of the joints of one hand (left or right hand) of the human (operator). The control device 2 also receives signals from one tracker 4 that represent the position and posture (orientation) of one hand (left or right hand) of the human (operator). The control device 2 controls one robot arm 7 based on these signals.

Alternatively, the control device 2 may control two robotic arms 7 in response to the movements of both hands of a human (operator). In this case, the control device 2 basically controls the movement of one of the two robotic arms 7 (the robotic arm corresponding to the right hand) based on operation with the human's (operator's) right hand. The control device 2 also controls the movement of the other robotic arm 7 (the robotic arm corresponding to the left hand) based on operation with the human's (operator's) left hand. The control device 2 receives signals from each of the two gloves 3 indicating the bending angles of the joints of one of the human's (operator's) hands (left hand or right hand). The control device 2 also receives signals from each of the two trackers 4 indicating the position and posture (orientation) of one of the human's (operator's) hands (left hand or right hand). The control device 2 controls the two robotic arms 7 based on these signals.

Glove 3 is a device that can be worn on the hand of a human (operator). Glove 3 is equipped with multiple angle sensors that detect the bending angles of the joints of the human hand wearing Glove 3 and output that angle information. Human fingers are made up of many joints. One Glove 3 detects the bending angles of a total of 15 joints for one human hand (right or left hand), for example, the DIP joint (first joint), MP joint (third joint), and CM joint (fourth joint) of the thumb; the DIP joint (first joint), PIP joint (second joint), and MP joint (third joint) of the index finger; the DIP joint (first joint), PIP joint (second joint), and MP joint (third joint) of the middle finger; the DIP joint (first joint), PIP joint (second joint), and MP joint (third joint) of the ring finger; and the DIP joint (first joint), PIP joint (second joint), and MP joint (third joint) of the little finger, and outputs that information. Angles are expressed as numerical values in units of degrees or radians, for example. When gloves 3 are used for both hands, the pair of gloves 3 detects the bending angles of a total of 30 joints and outputs that information. The gloves 3 pass signals representing information about the angles of multiple joints to the control device 2. In other words, the gloves 3 pass angle information expressed as, for example, a 15-dimensional vector for each hand to the control device 2.

In other words, the glove 3 transmits signals representing information about the bending angles of the joints of each finger on the user's hand to the input device interface unit 21 of the control device 2. The glove 3 may detect information about the bending angles of all joints in the user's hand and pass signals representing information about the bending angles of all of the joints to the input device interface unit 21.

As an example, MANUS Prime II series gloves can be used as gloves 3. However, other products may also be used as gloves 3.

The tracker 4 is a device that detects its own position and orientation and outputs this information. In this embodiment, one tracker 4 is attached to one glove 3. That is, one tracker 4 detects the position and orientation of the hand wearing the glove 3 and outputs that information. When gloves 3 are used on both hands, the tracker 4 outputs position and orientation information for each hand wearing the glove 3. The position is a position in a coordinate system of the X, Y, and Z axes (e.g., a Cartesian coordinate system) and is expressed as a three-dimensional vector. The orientation is, for example, the amount of rotation around each of the X, Y, and Z axes and is expressed as a three-dimensional vector. In other words, the tracker 4 passes position and orientation information, expressed as a six-dimensional vector, for each hand to the control device 2.

In other words, the tracker 4 transmits signals representing information about the position of the user's hand and information about the posture of the user's hand to the input device side interface unit 21 of the control device 2.

As an example, the VIVE Tracker from HTC CORPORATION can be used as the Tracker 4. However, other products may also be used as the Tracker 4.

The robot arm 7 is controlled by the control device 2. The robot arm 7 has arms and joints that correspond to a human arm, which allows it to change the position and posture (orientation) of the parts that correspond to a human hand. The robot arm 7 also has joints that correspond to the joints of a human hand (finger joints), and the bending angle of each joint can be changed independently. As mentioned above, there may be one or two robot arms 7. When there are two robot arms 7, the two robot arms 7 are controlled by the control device 2 based on the movements of both hands (left and right hands) of the human (operator). The robot arm itself is realized using existing technology.

The robot arm 7 has a function of feeding back its own state to the control device 2. That is, the robot arm 7 detects the position and posture (direction) of the parts corresponding to the hand and the bending angles of the joints of the parts corresponding to each finger. The robot arm 7 then feeds back this information (information expressed as an output vector V _Ok , described later) to the control device 2 at the necessary timing. The robot arm 7 feeds back this information, and the control device 2 performs control based on the feedback information from the robot arm 7, thereby achieving synchronization of the movements between the input device side (glove 3 and tracker 4) and the output device side (robot arm 7).

In other words, the robot arm 7 operates based on the control signal transmitted from the output device-side interface unit 23 of the control device 2, and transmits the status information of its own device to the output device-side interface unit at the specified timing.

As an example, products such as the UR3e, UR5e, UR10e, UR16e, and UR20 manufactured by Universal Robots can be used as the robot arm 7. However, other products may also be used as the robot arm 7.

The communication line 9 enables the exchange of information between the control device 2 and the robot arm 7. The communication line 9 may be, for example, a dedicated line provided for a specific user, or a public line (e.g., the Internet) shared by unspecified users. The existence of the communication line 9 makes it possible for the control device 2 to control the robot arm 7 even when the control device 2 and the robot arm 7 are located remotely from each other.

The input device side interface unit 21 of the control device 2 receives signals representing the position and orientation of the hand from the tracker 4, and receives signals representing the bending angles of the joints of each finger held by the hand from the glove 3. The input device side interface unit 21 passes the information (numerical information) represented by these received signals to the control function unit 22.

In other words, the input device interface unit 21 acquires signals representing information on the position of the user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger on the user's hand. The input device interface unit 21 may also acquire signals representing information on the bending angles of all joints in the user's hand from the glove 3.

The control function unit 22 of the control device 2 generates and outputs signals for controlling the robot arm 7 based on information (numerical information) received from the input device side interface unit 21. The control function unit 22 passes the generated signals for controlling the robot arm 7 to the output device side interface unit 23. In this embodiment, the control signals are, for example, signals that represent the speed of operation of each part of the robot arm 7.

In other words, the control function unit 22 generates and outputs a control signal for controlling the movement of the robot arm 7 corresponding to the user's hand, based on the position, posture, and bending angles of the joints of the user's hand. Furthermore, the control function unit 22 generates a control signal at a predetermined timing (the timing corresponding to when the output device side interface unit 23 receives state information from the robot arm 7), based on the state information passed from the output device side interface unit 23, so that the state of the user's hand (i.e., the state of the input device) and the state information of the robot arm 7 are consistent. This method of consistency (synchronization) will be described in more detail later. When the input device side interface unit 21 acquires signals representing information on the bending angles of all the joints in the user's hand from the glove 3, the control function unit 22 may generate and output a control signal based on the information on the bending angles of all of these joints.

The output device side interface unit 23 of the control device 2 receives the control signal (a signal for controlling the robot arm 7) generated by the control function unit 22 and transmits it to the robot arm 7. The output device side interface unit 23 may transmit the control signal to the robot arm 7 via the communication line 9. In other words, the output device side interface unit 23 transmits the control signal generated by the control function unit 22 to the robot arm 7. The output device side interface unit 23 also receives status information on the robot arm 7 at a predetermined timing (for example, every certain number of frames, but this timing is not limited to this) and passes it to the control function unit 22. The status information on the robot arm 7 is information that represents the hand position, hand posture, and bending angle of each finger joint on the robot arm 7 side, which correspond to the hand position, hand posture, and bending angle of the joint on the input device side.

Figure 2 is a schematic diagram showing the appearance of glove 3. As shown, glove 3 has a shape suitable for wearing on a human hand. The glove 3 shown in Figure 2 is for wearing on a human right hand, but a glove 3 with a symmetrical shape for wearing on a human left hand may also be implemented. The glove 3 shown in Figure 2 is made of materials such as cloth (fiber), synthetic resin, or rubber, and its shape can flexibly change in response to the bending of the joints in the human hand. The glove 3 also has sensors for detecting the bending angle of each joint in the human hand. A tracker 4 can be attached to the back of the glove 3. However, the tracker 4 may also be attached to a part of the glove 3 other than the back of the hand.

Figure 3 is a schematic diagram showing an example of the appearance of tracker 4. Tracker 4 is a small device that can be attached to glove 3, for example, as described above. By attaching it to glove 3, tracker 4 can output numerical values that represent the position (coordinate values on three-dimensional coordinate axes) and posture (amount of rotation around three-dimensional axes) of the hand of the person (operator) wearing glove 3.

Figure 4 is a schematic diagram showing an example of the appearance of the robot arm 7. As shown in the figure, the base of the robot arm 7 can be fixed to a stand or the like for use. The robot arm 7 also has multiple arms and joints. A robot hand can also be attached to the tip of the robot arm 7.

Figure 5 is a block diagram showing a more detailed functional configuration of the control function unit 22 described above. As shown in the figure, the control function unit 22 is composed of a control information generation unit 220, a synchronization control function unit 221, a noise correction function unit 222, and a motion capture teaching function unit 225.

The control information generation unit 220 generates and outputs a control signal for controlling the movement of the robot arm corresponding to the human (operator, user) hand based on the hand position, hand posture, and joint bending angle passed from the input device side interface unit 21.

The control information generation unit 220 generates a control signal at a predetermined timing based on the status information passed from the output device interface unit 23 (status information of the robot arm 7 passed as feedback information from the robot arm 7) so that the status of the human (operator, user) hand (i.e., the status detected by the input device (glove 3 or tracker 4)) is consistent with the status information of the robot arm 7. The consistency between these two is achieved by synchronization based on the status information that is fed back. The synchronization control function unit 221 performs the processing for this synchronization, and the control information generation unit 220 generates control information based on the results of the synchronization processing.

Based on the above feedback information (status information returned from the robot arm 7), the synchronization control function unit 221 performs processing to align the status of the human (operator, user) hand (i.e., the status detected by the input device (glove 3 or tracker 4)) with the status information on the robot arm 7.

The noise correction function unit 222 corrects noise contained in information passed from the input device (glove 3 or tracker 4). Specifically, the noise correction function unit 222 corrects velocity information calculated based on information passed from the input device (glove 3 or tracker 4) if the velocity information has a singular value.

Specifically, the noise correction function unit 222 determines whether noise is detected based on the temporal progression (i.e., speed information) of at least one of the following information: information on the position of the user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger on the user's hand. If noise is detected in any of the information, the noise correction function unit 222 corrects one of the information that caused the noise: information on the position of the user's hand, information on the posture of the user's hand, or information on the bending angles of the joints of each finger on the user's hand, so as to remove or mitigate the detected noise (for example, by taking the average value with the speeds in nearby frames, as described below).

Details of the noise correction process performed by the noise correction function unit 222 will be explained later with reference to Figures 7 and 8.

The motion capture teaching function unit 225 receives and stores the time series of control signals generated by the control function unit 22. When requested, the motion capture teaching function unit 225 reads the stored time series of control signals and controls the output device side interface unit 23 to transmit the time series of control signals to the robot arm 7.

The motion capture teaching function unit 225 is also called the "capturing function unit." The motion capture teaching function unit 225 achieves highly accurate teaching. For example, the motion capture teaching function unit 225 enables highly flexible and highly accurate teaching through a few key inputs by the user. The motion capture teaching function unit 225 may also record and list multiple teachings (time series of control information). By listing them, the user (operator) can select and execute the desired motion.

Figure 6 is a schematic diagram illustrating the principle by which the control function unit 22 controls the robot arm 7. Here, we will explain how the control function unit 22 controls the robot arm 7 with reference to Figure 6.

FIG. 6(A) shows a series of input vectors. FIG. 6(B) shows a series of vectors representing the state of the robot arm 7 (output side). VI0, _VI1 , _VI2 , _VI3 , ... (and so on) in FIG. 6(A) are the series of input vectors. _VO0 , _VO1 , _VO2 , _VO3 , _VO4 , ... (and so on) in FIG. 6(B) are the series of output vectors (vectors representing the state of the robot arm 7). Note that _VI0 , _VI1 , _VI2 , _VI3 , ... in FIG. ₆ (A) and _VO0 , _VO1 , _VO2 , _VO3 , _VO4 , ... in FIG. 6(B) are multidimensional vectors as described below, but are shown projected into two-dimensional space for convenience.

Here, the input vector series V _I0 , V _I1 , V _I2 , V _I3 , ... may be, for example, a series of numerical values (vectors) output by the tracker 4. Furthermore, the input vector series V _I0 , V _I1 , V _I2 , V _I3 , ... may be vectors representing the output from one tracker 4 (corresponding to either the left or right hand) or vectors representing the output from two trackers 4 (corresponding to both the left and right hands). In other words, in this case, each of V _Ik (k = 0, 1, 2, ...) is a six-dimensional vector (the position (three dimensions) and orientation (three dimensions) for only either the left hand or the right hand) or a twelve-dimensional vector (the position and orientation of both the left and right hands).

Alternatively, the input vector series V _I0 , V _I1 , V _I2 , V _I3 , ... may be a series of numerical values (vectors) output by both the tracker 4 and the glove 3. In this case, the input vector series V _I0 , V _I1 , V _I2 , V _I3 , ... may be vectors corresponding to only one hand (left hand or right hand) or both hands (left hand and right hand). In other words, in this case, each of V _Ik (k=0, 1, 2, ...) is a 21-dimensional vector (the position (three dimensions), posture (three dimensions), and bending angles of 15 joints of only either the left hand or the right hand) or a 42-dimensional vector (the above 21 dimensions for each of the left hand and the right hand, for a total of 42 dimensions).

However, the number of dimensions of the vector V _Ik (k=0, 1, 2, ...) does not necessarily have to be as described above. That is, for example, information on the bending angles of some joints may be missing, or information other than the information of the above number of dimensions may be included.

Furthermore, the output vector series V _O0 , V _O1 , V _O2 , V _O3 , V _O4 , ... may have the same number of dimensions as the input vector series V _I0 , V _I1 , V _I2 , V _I3 , .... Note that the space represented by the input vector V _Ik (including the coordinate system of the three-dimensional space output by the tracker 4) and the space represented by the output vector V _Ok (including the coordinate system of the three-dimensional space in which the robot arm 7 operates) do not necessarily have to be the same. In other words, the input vector and the output vector may be expressed as values in different coordinate systems. In this case, the control function unit 22 appropriately performs bidirectional conversion of coordinate values, etc. between the two coordinate systems.

The control function unit 22 can generate the input vector V _Ik based on information acquired from the input device (the glove 3 or the tracker 4) and calculations it performs itself. The control function unit 22 can also acquire the output vector V _Ok from the output device (the robot arm 7) as needed. However, as will be described later, reducing the frequency with which the control function unit 22 acquires information about the output vector V _Ok from the output device (the robot arm 7) reduces the processing load on the control device 2 itself and the communication load on the communication line 9.

The control function unit 22 controls the robot arm 7 at a fixed frame period. This frame period is designated Δt. Δt may be, for example, Δt≦22.300 ms (milliseconds). As an example, Δt = 22.222 ms. When Δt = 22.222 ms, the frame rate (1/Δt) is 45.000 fps (frames per second), with three decimal places. In other words, the frame rate is approximately 45 fps.

The shorter the frame period Δt for control, the more accurate the control by the control function unit 22. On the other hand, the shorter the frame period Δt, the more computational resources the control device 2 requires for control. Furthermore, if the frame period Δt is made too small, the glove 3, tracker 4, and robot arm 7 will be required to have high-speed responsiveness, resulting in higher device costs. Furthermore, to reduce the frame period Δt, the communication line 9 will also need to have specifications such as high capacity and low latency, which will increase costs. A frame rate of approximately 45 fps (Δt = 22.222 ms) as mentioned above, or a frame rate in the range of 40 fps to 50 fps, can achieve both high control accuracy and low-cost system implementation.

Specifically, the control function unit 22 generates velocity information (velocity vector) to control the robot arm 7 and transmits this velocity information to the robot arm 7 via the output device side interface unit 23. The robot arm 7 operates in accordance with the velocity information (velocity vector) passed from the control function unit 22.

The control function unit 22 can control the robot arm 7 by appropriately using the following two types of velocity information (velocity vectors). The first velocity information and second velocity information are control information for controlling the robot arm 7.

The first velocity information is velocity information based only on information supplied from the input side device (the glove 3 or the tracker 4). In other words, the first velocity information is information for controlling the robot arm 7 only by feedforward. The first velocity information is s _k calculated by the following equation (1). Note that the velocity information s _k is calculated in a discrete time series.

That is, s _k , which is the k-th first speed information in the sequence, is the displacement of the current input vector V _Ik from the previous input vector V _I(k-1) divided by the frame period Δt. The number of dimensions of the vector s _k is the same as the number of dimensions of the input vector V _Ik .

The second velocity information is velocity information obtained using information fed back from the output device (robot arm 7). In other words, the first velocity information is information for controlling the robot arm 7 using a feedback element. The second velocity information is s _k obtained by the following equation (2):

The function f() is a function that converts the coordinate system of the input vector into the coordinate system of the output vector. The function f ^-1 () that appears in equation (2) is the inverse function of the above function f(), and is a function that converts the coordinate system of the output vector into the coordinate system of the input vector. In other words, s _k , which is the k-th second velocity information in the series, is the value obtained by dividing the displacement of the current input vector V _Ik from the input vector corresponding to the previous output vector V _O(k-1) by the frame period Δt. The number of dimensions of the vector s _k is the same as the number of dimensions of the input vector V _Ik . It may be assumed that the number of dimensions of the input vector and the number of dimensions of the output vector are the same.

The second speed information s _k is generated by the control information generating unit 220 based on the result of the synchronization process performed by the synchronization control function unit 221 described above.
(Control information)

The control function unit 22 may control the robot arm 7 by appropriately using the first speed information or the second speed information described above.

As an example, the control function unit 22 controls the robot arm 7 using the second velocity information only once per P1 frame (P1 is a positive integer and a parameter value that can be set in advance; for example, P1 may be 5, etc.) (for example, only when the frame number is divisible by the integer P1). In this case, the control function unit 22 controls the robot arm 7 using the first velocity information at other times (for example, when the frame number is not divisible by the integer P1). In this case, synchronization is achieved between the information represented by the input vector V _Ik and the information represented by the output vector V _Ok at predetermined time intervals (for example, every P1 frame).

Instead of relying on the above example, the information represented by the input vector V _Ik and the information represented by the output vector V _Ok may be synchronized in another manner, i.e., at a frequency that is not necessarily at regular time intervals.

Specifically, the control function unit 22 applies the function f() to the first velocity information or the second velocity information, thereby transmitting a control value (velocity information) in the coordinate system of the output vector to the robot arm 7. That is, the control function unit 22 transmits velocity information shown in the following equation (3) to the robot arm 7.

That is, the control function unit 22 converts the velocity information _sk into the coordinate system of the output vector using the function f(), and controls the robot arm 7 using the conversion result.

7 and 8 are graphs illustrating noise correction by the noise correction function unit 222 in the control function unit 22. The noise correction function unit 222 corrects velocity noise caused by noise that may be present in the input vector V _lk . FIG. 7 is a line graph showing the progress of velocity before correction by the noise correction function unit 222. FIG. 8 is a line graph showing the progress of velocity after correction by the noise correction function unit 222. Note that the noise correction function unit 222 does not correct the velocity of the entire input vector V _lk , but rather corrects the velocity for each element of the input vector V _lk (however, there may be elements for which velocity correction is not performed).

Here, when the m-th element of the input vector V _Ik is V _Ik ^(m) , the velocity s _k ^(m) calculated for the m-th element from the (k-1)th input and the k-th input is expressed by the following equation (4).

Fig. 7 is a line graph showing the time series of the velocity s _k ^(m) calculated by the above equation (4) for a specific m-th element of the input vector V _Ik . Note that the horizontal axis of Fig. 7 represents the frame number (discrete time) and the vertical axis represents the velocity s _k ^(m) . In Fig. 7, the unit of velocity is meters per second (m/sec).

The method by which the noise correction function unit 222 corrects noise contained in this speed is as follows. First, the noise correction function unit 222 determines whether the speed s _k ^(m) calculated this time (frame number k) is noise. Specifically, the noise correction function unit 222 determines whether or not the speed s k (m) is noise based on whether or not the absolute value of the current speed s _k ^(m) is P3 times or more the average of the absolute values of the most recent P2 speeds.

Note that P2 is a configurable parameter value and is a positive integer. As an example, P2=3 may be used. P3 is a configurable parameter value and is a real number greater than 1.0. For example, the value of P3 may be set appropriately within the range of 5.0≦P3≦10.0. That is, as an example, when P2=3 and P3=5.0, the noise correction function unit 222 determines that s k ^(m) contains noise if the absolute value of the current speed s _k (m) is 5.0 times or more the average of the absolute values of s _k-1 ^(m) , s _k -2 ( ^m) , and s _{k- 3} ^{(m) .}

If the current speed s _k ^(m) contains noise as a result of the above determination, the noise correction function unit 222 calculates the average of the absolute values of the most recent P2 speeds and the absolute value of the current speed (a total of (P2+1) absolute values of speeds), and forcibly corrects the speed s _{k (m) so that the absolute value of the current speed s k} ^(m) _becomes ^the absolute value of the calculated average value. Note that at this time, the noise correction function unit 222 corrects the sign (positive or negative) of s _k ^(m) so that it is the same as the original sign.

That is, the noise correction function unit 222 forcibly rewrites the value of the m-th element of the input vector V _Ik so that it becomes the value of s _k ^(m) after correction as described above.

The noise correction function unit 222 may perform the above noise correction on all elements (all m) of the input vector V _{Ik ,} or may perform the above noise correction on only some of the elements. As an example, the noise correction function unit 222 may perform the above noise correction only on the position information (each of the three-dimensional (X-axis, Y-axis, Z-axis) coordinate values; in the case of both hands, the three-dimensional coordinate values for each hand) output by the tracker 4 among the elements of the input vector V Ik. Furthermore, the noise correction function unit 222 may perform the above noise correction only on other sets of elements of the input vector V _Ik .

Fig. 8 is a line graph showing the time series of the velocity s _k ^(m) for the m-th element of the input vector V _Ik after noise correction has been performed by the noise correction function unit 222. Note that in Fig. 8 as well, the horizontal axis represents the frame number (discrete time) and the vertical axis represents the velocity s _k ^(m) . The unit of this velocity is meters per second (m/sec).

Comparing Figures 7 and 8, it can be seen that the noise appearing in the graph of Figure 7 (for example, the noise indicated by symbols N1, N2, and N3 in Figure 7) has been removed from the graph of Figure 8. In other words, even if the input device (tracker 4 or glove 3) generates peculiar noise, the noise correction function unit 222 can correct that noise, thereby stabilizing the control of the robot arm 7 by the control function unit 22.

FIG. 9 is a schematic diagram showing the operation of capturing and reproducing a motion by the motion capture teaching function unit 225. FIG. 9(A) shows the operation when capturing a motion. FIG. 9(B) shows the operation when reproducing the captured motion. Whether or not to capture a motion may be set when the control device 2 is operating. The captured motion may be reproduced, for example, based on a request from an external user. The "control signal" below is f(s _k ) described in equation (3) above. In other words, the control signal is a signal that represents information about the speed instructed to the robot arm 7 in each frame.

As shown in Figure 9 (A), during capturing, the control signal generated by the control information generation unit 220 is also passed to the motion capture teaching function unit 225. The motion capture teaching function unit 225 stores the time series of this control signal on a recording medium or the like.

Also, as shown in FIG. 9(B), the motion capture teaching function unit 225 reproduces the saved motion when requested. That is, the motion capture teaching function unit 225 reads the saved time series of control signals from a recording medium or the like, and controls the output device side interface unit 23 to transmit the time series of control signals to the robot arm 7. Note that in the example shown in FIG. 9(B), the motion capture teaching function unit 225 passes the read time series of control signals to the output device side interface unit 23 via the control information generation unit 220. Alternatively, the motion capture teaching function unit 225 may pass the read time series of control signals directly to the output device side interface unit 23 without passing through the control information generation unit 220.

In this way, the motion capture teaching function unit 225 can capture motion and reproduce the captured motion. Note that when reproducing the captured motion, the control function unit 22 does not receive feedback of status information from the robot arm 7.

Figure 10 is a flowchart showing the processing procedure performed by the control device 2. The processing procedure (control method) will be explained below in accordance with this flowchart.

In step S1, the control device 2 performs initialization for controlling the robot arm 7. The initialization processing in this step includes processing to align (initial synchronization) the position, posture, and joint bending angles on the input device (gloves 3, tracker 4) with the position, posture, and joint bending angles on the output device (robot arm 7). In addition, in this step, the control device 2 may perform other initialization processing for controlling the robot arm 7.

The following steps S2 to S9 are processes for controlling each frame. The processing for each frame may be performed at a predetermined time interval (the aforementioned frame period Δt).

In step S2, the input device interface unit 21 of the control device 2 acquires the current vector information (the aforementioned input vector V _Ik ) from the input device (glove 3, tracker 4) side. The input device interface unit 21 passes the acquired input vector V _Ik to the control function unit 22. The control function unit 22 at least temporarily stores the passed input vector V _Ik .

In step S3, the control function unit 22 (noise correction function unit 222) performs a speed noise correction process. The speed noise correction process has already been described. The speed noise correction process may change some of the elements of the input vector V _Ik acquired in step S2. Note that if there is no speed noise, the correction process in this step does not produce any specific effect, and the input vector V _Ik acquired in step S2 is used as is.

In step S4, the control function unit 22 (synchronization control function unit 221) determines whether or not to synchronize the position, etc. with the robot arm 7 in this control. If synchronization with the robot arm 7 is to be performed (step S4: YES), the process proceeds to step S6. If synchronization with the robot arm 7 is not to be performed (step S4: NO), the process proceeds to step S5.

As mentioned above, the synchronization control function unit 221 may perform synchronization (step S4: YES), for example, every certain number of frames (in the example above, every P1 frames). It may also determine whether to perform synchronization based on other conditions.

If the process proceeds to step S5, in that step the control function unit 22 (control information generation unit 220) generates speed information (speed vector) s _k based on the current vector on the input device side (input side vector V _Ik ) and the previous vector on the input device side (input side vector V _I(k-1) ). In this step, the first speed information s _k is generated using the calculation according to the above-mentioned equation (1). After the processing of this step, the process proceeds to step S8.

If the process proceeds to step S6, in this step the output device side interface unit 23 acquires the previous vector information (output side vector V _O(k-1) ) from the output device (robot arm 7). The output device side interface unit 23 passes the acquired output side vector V _O(k-1) to the control function unit 22. The control function unit 22 stores the passed output side vector V _O(k-1) at least temporarily. After this step, the process proceeds to step S7.

In step S7, the control function unit 22 (control information generation unit 220) generates velocity information (velocity vector) s _k based on the current vector on the input device side (input side vector V _Ik ) and the previous vector on the output device side (output side vector V _O(k-1) ). In this step, second velocity information s _k is generated using the calculation according to the above equation (2). Note that the processing in this step synchronizes and aligns the information on the input device side with the information on the output device side (robot arm 7 side). Synchronization control is performed by the synchronization control function unit 221. After processing in this step, the process proceeds to step S8.

Next, in step S8, the control function unit 22 (control information generation unit 220) controls the robot arm 7 using the velocity information (velocity vector) s _k created up to step S7. That is, the control function unit 22 (control information generation unit 220) controls the robot arm 7 by transmitting the velocity information f(s _k ) shown in the above equation (3) to the robot arm 7. The robot arm 7 receives this velocity information f(s _k ) and operates in accordance with the velocity information f(s _k ).

In step S9, the control device 2 determines whether or not to terminate control. The determination in this step results in termination of control when, for example, a user (operator, etc.) issues a command to terminate the robot arm 7, when the operating period of the robot arm 7 ends according to a predetermined schedule, or when the occurrence of some kind of abnormality makes it impossible or difficult to continue operating the robot arm 7. If control is to be terminated (step S9: YES), the entire processing of this flowchart ends. If control is not to be terminated (step S9: NO), the process returns to step S2 to execute control for the next frame.

Next, several modified examples of the first embodiment will be described. Note that multiple modified examples may be combined to the extent possible.

[Modification 1]
In the first embodiment, the control function unit 22 has a motion capture teaching function unit 225 (see FIG. 5). As a modified example, the control function unit 22 does not have to have the motion capture teaching function unit 225. In this case, it is not possible to capture motion or reproduce captured motion, but other functions of the control device 2 are realized.

[Modification 2]
In the first embodiment, it was assumed that one glove 3 detects the bending angles of all the joints in one hand (left hand or right hand) of the user. As a modification, the glove 3 may not detect the bending angles of some joints. In this case, the amount of information that the glove 3 passes to the control device 2 is relatively reduced. Furthermore, the control device 2 cannot perform control according to the bending angles of the joints that were not detected. However, the control device 2 maintains the other functions that it had in the first embodiment.

[Modification 3]
In the first embodiment, the control function unit 22 has the noise correction function unit 222 (see FIG. 5 ). As a modified example, the control function unit 22 may not have the noise correction function unit 222. In this case, noise in the input information cannot be corrected, but other functions of the control device 2 are still realized.

Second Embodiment
Next, a second embodiment of the present invention will be described. Note that the following description may omit the matters already described in the previous embodiment. Here, the description will focus on matters unique to this embodiment.

Figures 11 and 12 are functional block diagrams showing the general functional configuration of a control device with a minimum configuration according to the second embodiment. As shown in Figure 11, the control device 1002 of this embodiment includes an input device interface unit 21, a control function unit 1022, and an output device interface unit 23. The functions of the input device interface unit 21 and the output device interface unit 23 are the same as those described in the first embodiment. The control function unit 1022 does not have some of the functions of the control function unit 22 in the first embodiment. As shown in Figure 12, the control function unit 1022 of this embodiment includes a control information generation unit 1220 and a synchronization control function unit 1221. The control function unit 1022 of this embodiment does not have the functions of the noise correction function unit 222 and the motion capture teaching function unit 225 that the control function unit 22 in the first embodiment had. In other words, the control function unit 1022 does not have the function of correcting noise contained in signals input from the input device side. Furthermore, the control function unit 1022 does not store the time series of control signals representing motion, nor does it perform playback based on the time series of stored control signals.

In this embodiment, too, the control function unit 1022 receives signals from the input device (glove or tracker) and generates and outputs control signals for controlling the movement of the robot arm corresponding to the hand based on the hand position, hand posture, and joint bending angle. Furthermore, the control function unit 1022 generates the control signals at predetermined timing based on the robot arm status information passed from the output device interface unit 23 so that the status of the user's hand matches the status information of the robot arm.

[Realization by computer and program]
FIG. 13 is a block diagram showing an example of the internal configuration for realizing the functions of each device in the first and second embodiments. At least some of the functions of each device (control device 2 or 1002, glove 3, tracker 4, and robot arm 7) can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input/output port 903, input/output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input/output port 903 is a port through which the central processing unit 901 exchanges data with external input/output devices, etc. The input/output devices 904 and 905 are input/output devices. The input/output devices 904 and 905 exchange data with the central processing unit 901 via the input/output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input/output port 903 via the bus 906.

At least some of the functions of each device in the embodiments (control device 2 or 1002, glove 3, tracker 4, and robot arm 7) can be implemented by a computer and a program. In this case, the program for implementing this function may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as floppy disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines when transmitting programs via networks like the Internet or communication lines like telephone lines, or media that store programs for a certain period of time, such as volatile memory within the computer systems that serve as the server or client in such cases. Furthermore, the above program may be designed to realize some of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded on the computer system.

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment and includes designs that do not deviate from the gist of the present invention.

As described above, according to any of the embodiments, the control function unit (22, 1022) generates a control signal at a predetermined timing based on the status information (i.e., status information representing the operating status of the robot arm (7)) passed from the output device interface unit (23) so that the status of the user's hand (i.e., the status of the input device (glove (3) or tracker (4))) is consistent with the status information of the robot arm (7). This allows the control function unit (22, 1022) to control the robot arm (7) with high precision. In other words, it is possible to synchronize the status (position, posture, joint bending angle, etc.) between the input device and the output device. This synchronization may be performed, for example, at regular time intervals or at other times.

In addition, the noise correction function unit (222) can correct noise that may occur on the input device (glove (3) or tracker (4) side). This means that control by the control function unit (22) is stable. In other words, noise caused by disturbances during the input stage can be corrected.

The input device side interface unit (21) acquires signals representing information on the bending angles of all joints in the user's (operator's) hands. The control function unit (22, 1022) generates and outputs control signals based on the information on the bending angles of all joints. This allows the control function unit (22) to control the robot arm (7) faithfully according to the bending angles of all joints in the user's (operator's) hands.

According to this embodiment, even when a robot control system is implemented by combining general-purpose items (such as gloves, trackers, and robot arms), highly accurate robot arm control is possible. In other words, an inexpensive, highly accurate robot control system can be implemented. In other words, highly accurate control that matches the intuition of the user (operator) can be achieved.

Some or all of the above embodiments may also be described as, but are not limited to, the following notes:

(Appendix 1)
an input device interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand;
a control function unit that generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint;
an output device side interface unit that transmits the control signal generated by the control function unit to the robot arm, and receives status information of the robot arm at a predetermined timing and passes the status information to the control function unit;
Equipped with
the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints,
the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other.
Control device.

(Appendix 2)
The control function unit
a noise correction function unit that, when detecting noise based on a temporal transition of at least any one of information regarding the position of the user's hand, information regarding the posture of the user's hand, and information regarding the bending angles of the joints of the fingers of the user's hand, corrects any one of the information regarding the position of the user's hand, information regarding the posture of the user's hand, and information regarding the bending angles of the joints of the fingers of the user's hand, which caused the detected noise, so as to eliminate or mitigate the detected noise;
2. The control device of claim 1, comprising:

(Appendix 3)
the input device side interface unit acquires signals representing information on bending angles of all joints in the user's hand;
The control function unit generates and outputs the control signal based on information on bending angles of all the joints.
3. The control device according to claim 1 or 2.

(Appendix 4)
a capturing function unit that stores a time series of the control signal generated by the control function unit, and, when requested, reads out the stored time series of the control signal and controls the output device side interface unit to transmit the time series of the control signal to the robot arm;
4. The control device according to claim 1, further comprising:

(Appendix 5)
an input device interface unit acquiring signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand;
a step in which a control function unit generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint;
an output device side interface unit transmitting the control signal generated by the control function unit to the robot arm, and receiving status information of the robot arm at a predetermined timing and passing the information to the control function unit;
Including,
the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints,
the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other.
Control method.

(Appendix 6)
an input device interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand;
a control function unit that generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint;
an output device side interface unit that transmits the control signal generated by the control function unit to the robot arm, and receives status information of the robot arm at a predetermined timing and passes the status information to the control function unit;
Equipped with
the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints,
the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other.
A program that makes a computer function as a control device.

(Appendix 7)
A control device according to any one of Supplementary Note 1 to Supplementary Note 4;
a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device;
a glove that transmits a signal representing information on the bending angle of each finger joint of the user's hand to the input device side interface unit of the control device;
A robot control system comprising:

(Appendix 8)
A control device according to any one of Supplementary Note 1 to Supplementary Note 4;
a robot arm that operates based on the control signal transmitted from the output device-side interface unit of the control device and transmits the status information of its own device to the output device-side interface unit at the predetermined timing;
A robot control system comprising:

(Appendix 9)
A control device according to any one of Supplementary Note 1 to Supplementary Note 4;
a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device;
a glove that transmits a signal representing information on the bending angle of each finger joint of the user's hand to the input device side interface unit of the control device;
a robot arm that operates based on the control signal transmitted from the output device-side interface unit of the control device and transmits the status information of its own device to the output device-side interface unit at the predetermined timing;
A robot control system comprising:

(Appendix 10)
The information on the position of the user's hand is expressed as three-dimensional coordinate values for one hand,
The user's hand posture information is expressed as a three-dimensional rotation angle for one hand.
5. The control device according to any one of claims 1 to 4.

(Appendix 11)
The control signal generated by the control function unit is a signal representing information on the speed of a portion of the robot arm.
11. The control device of claim 1, wherein the first and second inputs are connected to a first input port.

(Appendix 12)
the control function unit generates and outputs the control signal for each frame at a fixed time interval;
The predetermined timing is a timing for every fixed number of frames.
12. The control device according to any one of claims 1 to 4 and claims 10 to 11.

(Appendix 12)
The frame frequency is 45 frames per second (45 fps) or greater.
13. The control device of claim 12.

The present invention can be used, for example, in industries that use robots. Examples include controlling robots in manufacturing, remote surgery (or businesses that provide equipment for such surgery), and online tours, but its applicability is not limited to these industries. Furthermore, the scope of use of the present invention is not limited to the examples given here.

1 Robot control system 2 Control device 3 Glove 4 Tracker 7 Robot arm 9 Communication line 21 Input device side interface unit 22 Control function unit 23 Output device side interface unit 220 Control information generation unit 221 Synchronous control function unit 222 Noise correction function unit 225 Motion capture teaching function unit (capturing function unit)
901 Central processing unit 902 RAM
903 Input/output ports 904, 905 Input/output devices 906 Bus 1002 Control device 1022 Control function unit 1220 Control information generation unit 1221 Synchronous control function unit

Claims (9)

an input device interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand;
a control function unit that generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint;
an output device side interface unit that transmits the control signal generated by the control function unit to the robot arm, and receives status information of the robot arm at a predetermined timing and passes the status information to the control function unit;
a capturing function unit that stores a time series of the control signal generated by the control function unit, and, when requested by a user, reads out the stored time series of the control signal and controls the output device side interface unit to transmit the time series of the control signal to the robot arm;
Equipped with
the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints,
the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other.
Control device. the capturing function unit lists time series of the plurality of control signals, and enables a user to select a desired time series of the control signal from the list and execute it;
The control device according to claim 1 . The control function unit
a noise correction function unit that, when detecting noise based on a temporal transition of at least any one of information regarding the position of the user's hand, information regarding the posture of the user's hand, and information regarding the bending angles of the joints of the fingers of the user's hand, corrects any one of the information regarding the position of the user's hand, information regarding the posture of the user's hand, and information regarding the bending angles of the joints of the fingers of the user's hand, which caused the detected noise, so as to eliminate or mitigate the detected noise;
The control device of claim 1 , comprising: the input device side interface unit acquires signals representing information on bending angles of all joints in the user's hand;
The control function unit generates and outputs the control signal based on information on bending angles of all the joints.
The control device according to claim 1 . an input device interface unit acquiring signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand;
a step in which a control function unit generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint;
an output device side interface unit transmitting the control signal generated by the control function unit to the robot arm, and receiving status information of the robot arm at a predetermined timing and passing the information to the control function unit;
a step in which a capturing function unit stores a time series of the control signal generated by the control function unit, and when requested by a user, reads out the stored time series of the control signal and controls the output device side interface unit to transmit the time series of the control signal to the robot arm;
Including,
the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints,
the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other.
Control method. an input device interface unit that acquires signals representing information on the position of a user's hand, information on the posture of the user's hand, and information on the bending angles of the joints of each finger of the user's hand;
a control function unit that generates and outputs a control signal for controlling a movement of a robot arm corresponding to the hand based on the position of the hand, the posture of the hand, and the bending angle of the joint;
an output device side interface unit that transmits the control signal generated by the control function unit to the robot arm, and receives status information of the robot arm at a predetermined timing and passes the status information to the control function unit;
a capturing function unit that stores a time series of the control signal generated by the control function unit, and, when requested by a user, reads out the stored time series of the control signal and controls the output device side interface unit to transmit the time series of the control signal to the robot arm;
Equipped with
the state information of the robot arm is information representing a hand position, a hand posture, and bending angles of each finger joint on the robot arm side, which correspond to the hand position, the hand posture, and bending angles of the joints,
the control function unit generates the control signal at the predetermined timing based on the state information passed from the output device side interface unit so that the state of the user's hand and the state information of the robot arm are consistent with each other.
A program that makes a computer function as a control device. The control device according to claim 1 ;
a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device;
a glove that transmits a signal representing information on the bending angle of each finger joint of the user's hand to the input device side interface unit of the control device;
A robot control system comprising: The control device according to claim 1 ;
a robot arm that operates based on the control signal transmitted from the output device-side interface unit of the control device and transmits the status information of its own device to the output device-side interface unit at the predetermined timing;
A robot control system comprising: The control device according to claim 1 ;
a tracker that transmits signals representing information on the position of a user's hand and information on the posture of the user's hand to the input device side interface unit of the control device;
a glove that transmits a signal representing information on the bending angle of each finger joint of the user's hand to the input device side interface unit of the control device;
a robot arm that operates based on the control signal transmitted from the output device-side interface unit of the control device and transmits the status information of its own device to the output device-side interface unit at the predetermined timing;
A robot control system comprising: