JP7577017B2 - CONTROL DEVICE, ROBOT SYSTEM, CONTROL METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a control device, a robot system, a control method, and a program.

Conventionally, robot systems have been proposed that accept the actions of an operator and control a robot in accordance with the accepted actions. For example, the robot system described in Patent Document 1 acquires first user posture information indicating the posture of a first user operating the robot, and acquires pre-change posture information indicating a pre-change posture before the posture of the robot is changed based on the first user posture information. The robot system determines a target posture different from the posture of the first user as the posture of the robot based on the first user posture information at the time when the robot is in the pre-change posture information and the pre-change posture indicated by the pre-change posture information. The robot system described in Patent Document 1 also acquires user information on which the first user posture information is based from the user device via a network.

Patent No. 6476358

In a robot system, having the robot follow the operator's movements without delay is important for improving work efficiency and reducing the operator's burden. However, it is impossible to completely eliminate delays in controlling the robot based on the operator's movements and in communicating operation information. On the other hand, delays in the robot's movements (tracking delays) make the operator feel uncomfortable, slowing down work speed and causing work to fail, resulting in reduced work efficiency.

The present invention has been made in consideration of the above points, and one of its objectives is to provide a control device, robot system, control method, and program that can improve the operability.

(1) The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention includes a trajectory prediction unit that determines a predicted trajectory of an effector of the robot from a current time to a predicted time after a predetermined predicted time based on operation information that indicates at least a robot's operation and operation information that indicates at least the position of an operator's hand, a control command generation unit that generates a control command based on the predicted trajectory, and an operation status estimation unit that estimates an operation status of the robot based on environmental information that indicates at least an operational environment of the robot and the operation information, wherein the trajectory prediction unit is a control device that determines the predicted time based on the operation status .

(2) Another aspect of the present invention is the control device according to (1) , further comprising a drive control unit that determines an amount of operation for the movement mechanism of the robot based on a target value of a displacement of the movement mechanism of the robot that provides a target position of the effector for each time of the predicted trajectory, and the movement status estimating unit:
The target gain may be determined based on the operating conditions.

(3) Another aspect of the present invention is the control device of (2) , wherein the drive control unit determines the manipulated variable by combining a first component based on a deviation between the displacement output value giving the current position of the effector and the target value and a first gain, and a second component based on the target value and a second gain, and the operation status estimation unit may determine the first gain and the second gain based on the operation status.

(4) Another aspect of the present invention is a control device of any of (1) to (3) , wherein the operation status estimation unit may further estimate the operation status based on operator information indicating a status of an operator operating the robot.

(5) Another aspect of the present invention may be a program for causing a computer to function as a control device according to any one of (1) to (3) .

(6) Another aspect of the present invention may be a robot system including the control device according to any one of (1) to (3) and the robot.

(7) Another aspect of the present invention is a control method in a control device, comprising : a first step of determining a predicted trajectory of an effector of the robot from a current time to a predicted time after a predetermined predicted time based on operation information indicating at least a robot operation and operation information indicating at least a position of an operator's hand; a second step of generating a control command based on the predicted trajectory; and a third step of estimating an operation status of the robot based on environmental information indicating at least an operating environment of the robot and the operation information, wherein the first step includes a step of determining the predicted time based on the operation status .

According to the above-mentioned configurations (1), (5), (6), and (7) , the effector of the robot is driven according to a control command generated based on the predicted trajectory of the effector from the current time to a predicted time after the current time, thereby reducing or eliminating the delay until the operation is reflected in the operation of the robot 20. Since the operability for the operator is improved, it is possible to achieve both improved work efficiency and reduced burden.

According to the above-mentioned configurations (1), (5), (6), and (7) , the predicted time is determined according to the robot's operating environment and the robot's operating status estimated from the operating status, so that the balance between the improvement of the operability and the accuracy of the position of the effector to be controlled is adjusted according to the operating status.

According to the above-mentioned configuration (2) , the contribution of the target value to the amount of operation of the operating mechanism is adjusted in accordance with the operating situation. Therefore, the sensitivity of the operation of the effector to the operation by the operator is adjusted in accordance with the operating situation.

According to the above-mentioned configuration (3) , the balance between the feedback term and the feedforward term is adjusted according to the operating situation. Therefore, the balance between the sensitivity and accuracy of the operation of the effector organ in response to the operation of the operator is adjusted according to the operating situation.

According to the above-mentioned configuration (4) , the operating status can be accurately estimated by further referring to the status of the operator, which further improves the work efficiency and reduces the work burden of the robot.

1 is a schematic block diagram illustrating a configuration example of a robot system according to a first embodiment. 2 is a block diagram showing an example of a functional configuration of a portion of a control device according to the first embodiment; FIG. FIG. 2 is a schematic block diagram illustrating an example of a hardware configuration of a control device according to the first embodiment. 5 is a flowchart illustrating an example of an operation control process according to the first embodiment. FIG. 11 is a schematic block diagram illustrating a configuration example of a robot system according to a second embodiment. FIG. 11 is a block diagram showing an example of a functional configuration of a portion of a control device according to a second embodiment. 13 is a flowchart illustrating an example of an operation control process according to the second embodiment.

First Embodiment
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic block diagram showing an example of the configuration of a robot system S1 according to this embodiment.
The robot system S1 is a control system that can control the operation of the robot 20 according to the operation of an operator who is a user. The robot system S1 is also a manipulation system that manipulates the operation of the robot 20 according to the operation. The robot 20 is equipped with an effector (also called an end-effector, robot hand, end effector, effector, etc.). The effector is a member that mechanically acts on and affects other objects. The effector has a plurality of fingers and is equipped with a mechanism that can grasp or release other objects. Each finger can be operated according to the movement of the corresponding finger of the operator. This allows various tasks to be realized.

The robot system S1 according to this embodiment estimates the motion status of the robot 20 based on environmental information indicating the motion environment of the robot 20 and operation information indicating the operation status. As described below, when estimating the motion status, the robot system S1 may also use drive state information acquired from the drive sensor 26 and gaze information of the operator. The robot system S1 generates a control command for operating the effector of the robot 20 to a target position based on characteristic parameters indicating control characteristics corresponding to the motion status. The robot system S1 controls the motion of the robot 20 based on the generated control command.
In the present application, the operation situation mainly includes the form of the operation performed by the robot 20, that is, the task or operation mode. The operation situation may include, for example, the type of work to be performed, the positional relationship with an object related to the work, the type or characteristics of the object, and the like as elements. The operation situation may be defined to include any element or a combination of any elements. Even if the operation information is common, the target position to which the effector of the robot 20 is moved may differ depending on the operation situation.

The robot system S1 includes a robot 20, a display device 50, an operating device 70, and an environmental information acquisition unit 80. The robot 20 includes one or more manipulators. The manipulator is also called a robot arm. Each manipulator includes a plurality of segments, which are connected to one another. Each of the plurality of segments includes one effector. The effector is a member that is connected to one end of the manipulator and that contacts an object to exert an action. The operating mechanism of the robot 20 includes a joint for each pair of segments including two of the plurality of segments, and an actuator for each joint. The position of the effector can be moved by changing the angle between each pair of segments using the actuator. In the following description, the case where the number of manipulators is one (single-arm type) is mainly described, but the number of manipulators may be two or more. The robot 20 may be a stationary type that is fixed to a predetermined position, or a mobile type that can move its own position. The following explanation focuses mainly on the case where the robot 20 is a stationary type.

The control device 30 of the robot 20 is connected to the display device 50, the operation device 70, and the environmental information acquisition unit 80 wirelessly or via wires so as to be able to input and output various types of information.
The display device 50 and the operation device 70 may be located in a space physically separated from the robot 20 and the environmental information acquisition unit 80. In that case, the display device 50 and the operation device 70 may be configured as a remote operation system connected to the control device 30 via a communication network.

The robot 20 includes a drive unit 24 , a drive sensor 26 , a power source 28 , and a control device 30 .
The drive unit 24 includes an actuator for each joint, and operates according to a control command input from a drive control unit 40 of the control device 30. Each actuator corresponds to a so-called motor, and changes the angle between the two segments connected to the drive unit 24 according to a target value of the drive amount specified by the control command.
The drive sensor 26 detects the drive state of the robot 20 by the drive unit 24, and outputs drive state information indicating the detected drive state to the control device 30. The drive sensor 26 includes, for example, a rotary encoder that detects the angle between two segments for each joint.

The power supply 28 supplies power to each component of the robot 20. The power supply 28 includes, for example, a power supply terminal, a secondary battery, and a voltage converter. The power supply terminal allows a power line to be connected, receives power from an external source, and supplies the supplied power to the secondary battery or the voltage converter.
The secondary battery stores power supplied via a power supply terminal. The secondary battery is capable of supplying power to each component via a voltage converter. The voltage converter converts the voltage of the power supply terminal or the power supplied from the power supply terminal to a predetermined voltage required by each component, and supplies each component with the converted voltage.

Next, the functional configuration of the control device 30 according to this embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of a portion of the control device 30 according to this embodiment. The control device 30 includes an information acquisition unit 32, an operation status estimation unit 34, a target position estimation unit 36, a control command generation unit 38, a drive control unit 40, a communication unit 42, and a storage unit 44.
The information acquisition unit 32 acquires various information related to the status of the operator and the status of the robot 20. For example, the information acquisition unit 32 acquires gaze information and first operator movement information from the display device 50. The gaze information and the first operator movement information constitute operator information indicating the status of the operator. The gaze information is information indicating the gaze direction of at least one eye of the operator at a certain time. The first operator movement information is information indicating the movement of the operator's head. The head movement is represented by the direction and position of the head for each time. In the present application, the gaze information and the first operator movement information may be collectively referred to as operator information.

The information acquisition unit 32 acquires second operator motion information from the operation device 70. The second operator motion information is information indicating body motion related to operation by the operator. The second operator motion information constitutes operation information indicating the operation status of the robot 20. The second operator motion information includes information mainly indicating hand motion as the body part of the operator. The hand motion includes motion of at least two fingers. The second operator motion information may include information indicating wrist motion in addition to hand motion. The wrist motion may be represented using a representative position of the wrist, and may further include posture information. The motion of the body part is represented using information on the position of the body part at each time. In this application, the second operator motion information may be referred to as operation information.

The information acquiring unit 32 acquires environmental information from the environmental information acquiring unit 80. The environmental information indicates the operating environment of the robot 20. The environmental information includes image data indicating an image showing the operating environment of the robot 20, and distance data indicating distances to objects distributed in the operating environment of the robot 20.
The information acquisition unit 32 acquires driving state information from the driving sensor 26. The driving state information can also be considered as posture information indicating the posture of the robot 20. The time series of the driving state information at each time corresponds to robot operation information indicating the operation of the robot 20. The information acquisition unit 32 stores the acquired various information in the storage unit 44.

The operation status estimation unit 34 reads the latest environmental information and operation information at that time from the storage unit 44, and estimates the operation status of the robot 20 based on the read environmental information and operation information. The operation status estimation unit 34 uses, for example, a predetermined machine learning model to estimate the operation status of the robot 20. When the environmental information and operation information are input as input information to the operation status estimation unit 34, a parameter set learned using training data is set in advance so that output information in which the reliability of a known operation status corresponding to the input information is 1 and the reliability of other candidate operation statuses is 0 is obtained as output information. The training data is composed of many pairs of input information and known output information. The parameter set is learned so that the difference between the estimated value of the output information estimated from the input information and the known output information is minimized as the entire training data. In that case, the reliability of each operation status obtained from the machine learning model can be determined as a real value between 0 and 1. The motion situation estimation unit 34 can identify the motion situation candidate for which the reliability calculated for each candidate motion situation is higher than a predetermined reliability and is the highest as the motion situation of the robot 20. As the motion situation, for example, the approach to an object (target object) that is located in the direction closest to the motion direction of the hand indicated by the operation information at that time among multiple objects located in the motion environment of the robot 20, the form of action of the object in contact with the effector at that time (e.g., grasping, release, friction, etc.) and the like are determined.

The motion situation estimation unit 34 may read the latest operator information at that time from the storage unit 44, and estimate the motion situation of the robot 20 based on the operator information in addition to the environmental information and the operation information. The operator information indicates the line of sight direction taking into account the movement of the operator's head. Therefore, when estimating the motion situation, the direction in which an event in which the operator is interested occurs is further taken into consideration based on the operator information. The motion situation estimation unit 34 pre-sets a parameter set learned as described above using training data so that, when the environmental information, operation information, and operator information are input as input information to the machine learning model, output information in which the reliability of a known motion situation corresponding to the input information is 1 and the reliability of other motion situation candidates is 0 is obtained as an output.
The operation status estimation unit 34 may read out the latest driving state information at that time from the storage unit 44, and further estimate the operation status of the robot 20 based on the driving state information. The driving state information indicates the posture of the robot 20 and its change over time. Therefore, when estimating the operation status, the posture of the robot 20 is further taken into consideration. In the operation status estimation unit 34, a parameter set learned as described above is set in advance using training data so that when driving state information is input as input information to the machine learning model in addition to environmental information and operation information, output information in which the reliability of a known operation status corresponding to the input information is 1 and the reliability of other operation status candidates is 0 is obtained as output.

The operation status estimation unit 34 determines characteristic parameters corresponding to the estimated operation status. The characteristic parameters are parameters related to the control characteristics for the operation of the effector. The characteristic parameters include, for example, a convergence determination parameter, a weighting coefficient for each factor of the objective function, or a combination of these. The memory unit 44 stores in advance a characteristic parameter table indicating characteristic parameters for each operation status. The operation status estimation unit 34 refers to the characteristic parameter table, identifies characteristic parameters corresponding to the estimated operation status, and outputs the identified characteristic parameters to the control command generation unit 38.

The convergence judgment parameter is a parameter indicating a convergence judgment condition for judging whether the position of the effector has converged to the target position. The strength of the convergence judgment condition corresponds to the degree of constraint of the converged position (sometimes referred to as the "convergence position" in this application) with respect to the target position, in other words, the accuracy with respect to the target position. For example, a threshold value of the distance from the target position can be used as the convergence judgment parameter. The smaller the threshold value of the distance from the target position, the stronger the convergence judgment condition. The operation situation estimation unit 34 may set a convergence judgment parameter indicating a stronger convergence judgment condition for an operation situation requiring tracking of the position of the effector to be controlled. Tracking means a characteristic that allows the position of the effector to be accurately controlled with respect to the target position, or a characteristic that requires accurate control, that is, accuracy. Such operation situations include, for example, a situation where a delicate object is installed in which the effector is sufficiently close to the target position, a situation where the operation instructed by the operation information is delicate, etc.

Inverse kinematics calculation is an analytical method for determining the displacement of each joint from a target position, that is, either or both of the angle and angular acceleration, and is also called inverse kinematics analysis. Note that inverse dynamics calculation is an analytical method for determining the driving force of a joint for realizing the motion of each joint (that is, the time change of either or both of the angle and angular acceleration) in a link mechanism in which multiple segments are connected, and is also called inverse dynamics analysis. Therefore, inverse kinematics calculation is closely related to inverse dynamics calculation. In this embodiment, characteristic parameters related to inverse kinematics calculation may include characteristic parameters related to inverse dynamics calculation.
On the other hand, the stronger the convergence judgment condition, the higher the possibility that no solution of the inverse kinematics calculation exists, or that even if a solution exists, the higher the possibility that the solution is unstable. Therefore, the motion situation estimation unit 34 may set a convergence judgment parameter indicating a weaker convergence judgment condition (e.g., distance from the target position) for a motion situation in which flexibility of the position of the effector to be controlled is permitted. Flexibility means a characteristic that allows deviation of the position of the effector from the target position. Flexibility means a characteristic that allows reliable control, that is, safety. An example of a motion situation in which flexibility is permitted is a case in which multiple objects are distributed around the path to the target position.

The objective function is used to numerically obtain the optimal solution for the operation of the robot 20 in the inverse kinematics calculation using a known method (optimization problem). In other words, the objective function is a function that quantitatively indicates the magnitude of the load for operating the effector to the target position, and includes multiple types of factors, and is composed of these factors. The weighting coefficient indicates the contribution of each factor to the objective function. The load means the cost related to control. The multiple factors include the magnitude of the control error at each time as one factor. The magnitude of the control error is expressed, for example, by the magnitude (distance) of the error between the final target position and the convergence. The larger the weighting coefficient for the control error, the higher the tracking ability of the effector to the target position is controlled. The objective function may include any one or a combination of factors, such as the arrival time from the current position of the effector to the convergence position, the magnitude of the jerk from the current position to the convergence position, the amount of power from the current position to the convergence position, the momentum of the robot 20 until the convergence position, the magnitude of the difference between the current posture of the robot 20 and the posture of the robot 20 at the time when the effector is located at the convergence position, etc. In this application, optimization means searching for a solution that reduces the function value of the objective function, and is not limited to absolutely minimizing the function value. Therefore, the function value may temporarily increase.

Therefore, the operation situation estimation unit 34 may set the weighting coefficient for the control error to be larger for an operation situation requiring more tracking of the position of the effector to be controlled. The operation situation estimation unit 34 may set the weighting coefficient for the control error to be smaller for an operation situation allowing more flexibility in the position of the effector to be controlled. The operation situation estimation unit 34 may set the weighting coefficient for the magnitude of the jerk to be larger for an operation situation requiring more continuity in the position of the effector to be controlled.

The motion situation estimating unit 34 determines drive control parameters corresponding to the estimated motion situation. The drive control parameters are parameters related to the control characteristics for driving each segment constituting the robot 20. The drive control parameters are used when a controller constituting the drive control unit 40 determines the operation amount for the actuator that drives each joint. A drive control parameter table indicating drive control parameters for each motion situation is stored in advance in the storage unit 44. The motion situation estimating unit 34 refers to the drive control parameter table to identify drive control parameters corresponding to the estimated motion situation, and outputs the identified drive control parameters to the drive control unit 40.
For example, when the controller is a PID (Proportional-Integral-Differential) controller, the drive control parameters include a proportional gain, an integral gain, and a differential gain. The proportional gain is a gain for multiplying a proportional term, which is one component of the manipulated variable, by the deviation between the target value and the output value at that time (present). The integral gain is a gain for multiplying an integral term, which is another component of the manipulated variable, by the integral value of the deviation up to that time. The differential gain is a gain for multiplying a differential term, which is yet another component of the manipulated variable, by the differential value of the deviation at that time. The operation status estimation unit 34 may determine each gain such that the differential gain is relatively larger than other types of gains, for example, in an operation status requiring more tracking ability. The operation status estimation unit 34 determines each gain such that the integral gain is relatively larger than other types of gains, for example, in an operation status requiring more flexibility.

The target position estimation unit 36 reads out the latest environmental information and operation information at that time, and estimates the target position of the effector of the robot 20 based on the read environmental information and operation information. The target position estimation unit 36 outputs target position information indicating the estimated target position to the control command generation unit 38.
The target position estimation unit 36 uses, for example, a machine learning model to estimate the target position. To distinguish it from the machine learning model in the operation situation estimation unit 34, the machine learning model in the operation situation estimation unit 34 and the machine learning model in the target position estimation unit 36 may be called a first machine learning model and a second machine learning model, respectively.

The target position estimation unit 36 is preset with a parameter set that has been learned so that when environmental information and operation information are input as input information to the machine learning model, output information related to the target position corresponding to the input information is obtained.
As the target position, a position on the surface of the object that is empirically likely to be acted on by the effector is set. The target position may depend on the shape of the object. In the case of a vertically elongated cylinder whose height is greater than its diameter, the position closest to the effector among the center of the bottom surface, the center of the surface, and the cross section of the midpoint in the height direction may be more likely to be the target position. The output information from the machine learning model may be, but is not limited to, information indicating the coordinates of the target position. The output information may include, for example, information indicating the object on which the target position is set, the type, shape, and direction of the object, and the position on the object where the target position is set. The target position estimation unit 36 can determine the coordinates of the target position from the output information.

The target position estimation unit 36 may also read the latest operator information at that time from the storage unit 44, and estimate the target position based on the operator information in addition to the environmental information and operation information. Therefore, the target position is estimated based on the operator information, taking into consideration the direction in which an event of interest to the operator occurred. However, a parameter set that has been learned so that when the operator information is input as input information to the machine learning model in addition to the environmental information and operation information, the coordinate value of the target position corresponding to the input information is obtained as output information is set in advance in the target position estimation unit 36.

However, the target position estimation unit 36 does not necessarily need to estimate the target position using the second machine learning model. For example, when the motion situation notified by the motion situation estimation unit 34 is a motion situation that does not involve continuous translational motion of the effector, the target position estimation unit 36 does not estimate the target position using the second machine learning model. Such motion situations include, for example, stillness, rotation, gripping and releasing an object, and friction on the surface of an object. When the target position estimation unit 36 does not estimate the target position using the second machine learning model, the target position estimation unit 36 may adopt the position of the operator's hand indicated by the operation information as the target position.
The target position estimation unit 36 can use mathematical models such as a neural network and a random forest as the first machine learning model and the second machine learning model.

The control command generating unit 38 generates an operation command for moving the effector from the current position toward the target position indicated by the target position information input from the target position estimating unit 36. "Towards the target position" or "towards the target position" includes the meaning that the purpose is to move to the target position, but reaching the target position is not guaranteed. The control command generating unit 38 can read the drive state information stored in the memory unit 44 and calculate the current position of the effector based on the angle of each joint and the dimension of each segment indicated by the drive state information. The control command generating unit 38 can calculate the speed of the effector based on the angle of each joint and the dimension of each segment indicated by the drive state information, as well as the angular velocity of each joint. The control command generating unit 38 performs a known inverse kinematic calculation based on the characteristic parameters and drive state information (robot operation information) input from the operation status estimating unit 34, and determines the angle of each joint so as to move the effector to the convergence position. The control command generating unit 38 outputs an operation command (joint command) indicating the determined angle of each joint (joint angle) to the drive control unit 40.

When the characteristic parameters include a convergence judgment parameter, the control command generation unit 38 may determine, in the inverse kinematics calculation, any position within the region from the target position indicated by the convergence judgment parameter as the convergence position and stop the movement of the effector 38. When determining whether the current position of the effector has converged, the control command generation unit 38 may, for example, determine whether the current position of the effector is within a predetermined distance threshold indicated by the characteristic parameter from the target position. When determining that the position of the effector has converged, the control command generation unit 38 generates a control command indicating the stop of operation and outputs it to the drive control unit 40. When the drive control unit 40 receives a control command indicating the stop of operation from the control command generation unit 38, it stops updating the target values for each joint and fixes them. At the convergence position, the control command generation unit 38 may impose a constraint condition that the velocity of the effector is zero, or the drive control unit 40 may impose a constraint condition that the angular velocity of each joint is zero.

When the convergence characteristic parameters include weighting coefficients for each factor of the objective function, the control command generator 38 performs optimization calculations based on the objective function synthesized using the factors corresponding to the weighting coefficients in the inverse kinematics calculation. The objective function is calculated, for example, as the sum (weighted sum) of factors that are the products of the factors and their weighting coefficients. In the optimization calculations, the control command generator 38 may recursively calculate the angles of each joint so that the objective function, which indicates the load generated until the effector reaches the target position or convergence position, is minimized.

When target position information is not acquired from the target position estimation unit 36, the control command generation unit 38 may adopt the position of the hand indicated by the operation information as the target position. The control command generation unit 38 performs inverse kinematic calculations to generate a motion command with the angle of each joint as the target value for moving the effector from the current position toward the adopted target position.

The drive control unit 40 determines the drive command based on the control command input from the control command generation unit 38, the drive control parameters input from the operation status estimation unit 34, and the drive state information (robot operation information) read from the storage unit 44. More specifically, the drive control unit 40 includes a controller that calculates the amount of operation for controlling the angle of each joint indicated by the control command as a target value, and determines the control command indicating the calculated amount of operation. The drive control unit 40 outputs the determined control command to the drive unit 24. The controller sets the angle indicated by the drive state information as an output value, and calculates the amount of operation by applying the drive control parameters to the deviation between the target value and the output value according to the control method set in the controller. The PID controller calculates the sum of the multiplication values obtained by multiplying the deviation, the time integral value of the deviation, and the time derivative value of the deviation by the proportional gain, the integral gain, and the derivative gain, respectively, as the amount of operation.

The communication unit 42 wirelessly or wiredly transmits and receives various information between each component of the robot 20 and other components (i.e., the display device 50, the operation device 70, and the environmental information acquisition unit 80). The communication unit 42 includes, for example, an input/output interface, a communication interface, and the like.
The storage unit 44 temporarily or permanently stores data used for various processes in the control device 30, various data acquired by the control device 30, etc. The storage unit 44 includes a storage medium such as a RAM (Random Access Memory) or a ROM (Read Only Memory).

The display device 50 includes a display unit 52, a gaze detection unit 54, a motion detection unit 56, a control unit 58, and a communication unit 60. The display device 50 may be equipped with a support member that can be attached to the head of the user (operator), and may be configured as a head mounted display (HMD).

The display unit 52 receives image data from the environmental information acquisition unit 80 via the control device 30 using the communication unit 60. Based on the received image data, the display unit 52 displays an image showing the operating environment of the robot 20. When the display unit 52 is worn on the head of the operator, the display unit 52 may be disposed in a position where the screen faces directly in front of both eyes.
The gaze detection unit 54 includes a gaze sensor that detects the gaze direction of one or both eyes of the operator. The gaze detection unit 54 transmits gaze information indicating the gaze direction detected at each time to the control device 30 using the communication unit 60. The gaze detection unit 54 may be disposed at a position exposed to at least one eye when worn on the operator's head.
The motion detection unit 56 detects a motion of the head of the operator, and transmits first operator motion information indicating the detected motion to the control device 30 using the communication unit 60. The motion detection unit 56 is configured to include, for example, an acceleration sensor for detecting the motion of the operator.

The control unit 58 controls the functions of the display device 50. The control unit 58 includes a processor such as a CPU (Central Processing Unit).
The communication unit 60 transmits and receives various information to and from the control device 30. The communication unit 60 includes a communication interface.

The operation device 70 includes a motion detection unit 72, a control unit 74, and a communication unit 76. The operation device 70 includes a support member that can be worn on the hand of an operator who is a user, and may be configured as a data glove.
The motion detection unit 72 detects the motion of the operator's hand, and transmits second operator motion information indicating the detected motion to the control device 30 using the communication unit 76. The motion detection unit 72 is configured to include, for example, an acceleration sensor for detecting the motion of the operator's hand. A wrist tracker may be further connected to the operation device 70. The wrist tracker is configured to include a support member that can be attached to the operator's wrist, and an acceleration sensor for detecting the wrist motion. The acceleration sensor that detects the wrist motion constitutes a part of the motion detection unit 72. The second operator motion information is transmitted to the control device 30 including information indicating the wrist motion.

The control unit 74 controls the functions of the operation device 70. The control unit 74 includes a processor such as a CPU.
The communication unit 76 transmits and receives various information to and from the control device 30. The communication unit 76 includes a communication interface.
The motion detection unit 72 may be connectable to the communication unit 60 of the display device 50. The second operator motion information may be transmitted to the control device via the communication unit 60. In that case, the control unit 74 and the communication unit 76 may be omitted.

The environmental information acquisition unit 80 includes an image capture unit 82, a distance measurement unit 84, and a communication unit 86. The environmental information acquisition unit 80 may be installed in the housing of the robot 20, or may be installed in a location separate from the robot 20.
The photographing unit 82 photographs an image of the operating environment within a predetermined range from the robot 20. The operating environment includes the range that the effector of the robot 20 can reach. The photographed image does not necessarily include the entire image of the robot 20. The photographing unit 82 is a digital video camera that photographs images at predetermined time intervals. The photographing unit 82 transmits image data showing the photographed image to the control device 30 via the communication unit 86.

The distance measuring unit 84 measures the distance from the robot 20 to the surface of an object in the operating environment within a predetermined range. The distance measuring unit 84 includes, for example, a wave transmitting unit, a wave receiving unit, and a distance detecting unit. The wave transmitting unit transmits waves such as infrared rays. The wave transmitting unit includes, for example, a light emitting diode. The wave receiving unit receives reflected waves generated by reflection on the surface of the object. The wave receiving unit includes, for example, a photodiode. The reflected waves are generated in response to the wave transmitted from the wave transmitting unit being incident on the surface of the object as an incident wave. The distance detecting unit detects the phase difference between the transmitted wave and the reflected wave, and can determine the distance to the surface of the object based on the detected phase difference. The distance detecting unit transmits distance data indicating the distance determined for each direction corresponding to each pixel of the image to the control device 30 via the communication unit 86. Note that the wave used to measure the distance is not limited to infrared rays, and may be millimeter waves, ultrasonic waves, or the like.
The communication unit 86 transmits and receives various information to and from the control device 30. The communication unit 86 includes a communication interface.
In addition, if the environmental information acquisition unit 80 is installed in the housing of the robot 20 and can send and receive various information with other functional units of the robot 20, the communication unit 86 may be omitted.

(Hardware configuration)
Next, an example of the hardware configuration of the control device 30 according to the present embodiment will be described.
3 is a schematic block diagram showing an example of a hardware configuration of the control device 30 according to the present embodiment. The control device 30 functions as a computer including a processor 102, a ROM 104, a RAM 106, an auxiliary storage unit 108, and an input/output unit 110. The processor 102, the ROM 104, the RAM 106, the auxiliary storage unit 108, and the input/output unit 110 are connected to each other so as to be able to input and output various data.

The processor 102, for example, reads out programs and various data stored in the ROM 104, and executes the programs to execute processes for realizing the functions of each part of the control device 30 and processes for controlling the functions. The processor 102 is, for example, a CPU.
In this application, the execution of processes instructed by various commands written in a program may be referred to as "executing a program" or "running a program."

The ROM 104 stores, for example, a program to be executed by the processor 102 .
The RAM 106 functions as a working area for temporarily storing various data and programs used by the processor 102, for example.
The auxiliary storage unit 108 permanently stores various types of data. The auxiliary storage unit 108 stores data acquired by the control device 30. The auxiliary storage unit 108 includes a storage medium such as a hard disk drive (HDD) or a solid state drive (SSD).

The input/output unit 110 is capable of wirelessly or wiredly inputting and outputting various types of data to and from other devices. The input/output unit 110 may be connected to other devices via a network. The input/output unit includes, for example, one or both of an input/output interface and a communication interface.
The display device 50, the operation device 70, and the environmental information acquisition unit 80 may also have a hardware configuration similar to the hardware configuration illustrated in FIG. 3, and may be configured as a computer that realizes the functions of each device.

(Operation Control Processing)
Next, an example of the operation control process according to the present embodiment will be described with reference to a flowchart shown in FIG.
(Step S102) The motion status estimation unit 34 acquires the latest environmental information, operation information, and gaze information at that time, and estimates the motion status of the robot 20 based on these.
(Step S104) The operation status estimation unit 34 refers to the characteristic parameter table and the drive control parameter table, and determines the characteristic parameters and the drive control parameters corresponding to the estimated operation status, respectively.

(Step S106) The target position estimation unit 36 acquires the latest environmental information, operation information, and gaze information at that time point, and estimates the target position of the effector of the robot 20 based on these.
(Step S108) The control command generator 38 generates a control command for moving the effector from the current position to the target position based on the determined characteristic parameters.
(Step S110) The drive control unit 40 uses the drive control parameters to generate drive commands for controlling the angles of the joints to target values indicated in the control commands.

(Step S112) The control command generator 38 determines whether the position of the effector has converged. The control command generator 38 can determine whether the position of the effector has converged, for example, by whether the current position of the effector is within a predetermined distance threshold indicated by the characteristic parameter from the target position. When it is determined that the position of the effector has converged (Step S112 YES), a control command indicating the stop of operation is generated and output to the drive control unit 40. This stops the operation of the robot 20. Then, the processing of FIG. 4 is terminated. When it is determined that the position of the effector has not converged (Step S112 NO), the processing returns to Step S102.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the following description, differences from the first embodiment will be mainly described, and functions and configurations common to the first embodiment will be referred to in the description of the first embodiment unless otherwise specified. The functional configuration of the robot system according to this embodiment will be described with reference to FIG. 5 as well as FIG. 6. FIG. 5 is a schematic block diagram showing an example of the configuration of the robot system S1 according to this embodiment. FIG. 6 is a block diagram showing an example of the functional configuration of a part of the control device 30 according to this embodiment.

The robot system S1 according to this embodiment determines a predicted trajectory of the effector of the robot 20 from the current time to a predicted time a predetermined predicted time later, based on position information indicating the position of the robot 20 and operation information indicating the operation status. The robot system S1 generates a control command based on the determined predicted trajectory. The robot system S1 according to this embodiment further includes a trajectory prediction unit 46 in the control device 30.

The trajectory prediction unit 46 predicts a predicted trajectory indicating the operation of the effector of the robot 20 from the present to a predicted time after a predetermined predicted time using robot operation information (driving state information) indicating at least the operation of the robot 20 at that time and operation information. The operation information is a factor in the combination with acceleration or deceleration of the effector, or a change in direction. The trajectory prediction unit 46 performs linear prediction on the driving state information and operation information within a predetermined period up to the present, for example, and calculates the position and speed of the effector up to the predicted time. The trajectory prediction unit 46 calculates a predicted trajectory based on the time series of the position of the effector at each time up to the predicted time. The predicted time indicates the elapsed time from the present to the predicted time, and is not the time spent for the prediction.

The trajectory prediction unit 46 may, for example, use a Kalman filter to sequentially predict the position of the effector. The Kalman filter has (a) a prediction step and (b) an update step, and is a method of sequentially repeating these steps. In the (a) prediction step, the trajectory prediction unit 46 adds the time evolution (weighted sum) of the state quantity of the effector up to the previous time and the external force to calculate an estimate of the current state quantity of the effector. However, the state quantity is a vector having position and velocity as elements. The external force is a vector having velocity and acceleration as elements. The velocity is calculated based on the angle and angular velocity of each joint indicated in the drive state information and the dimensions of the segment. The acceleration is calculated based on the angle, each velocity, and angular acceleration of each joint indicated in the drive state information and the dimensions of the segment. In addition, the trajectory prediction unit 46 adds the time evolution of the prediction error of the state quantity up to the previous time and the time evolution of the error to calculate the current prediction error. (b) In the update step, the trajectory prediction unit 46 calculates the difference between the current position (observation value) of the effector and the estimated value of the current position of the effector as the observation residual. The trajectory prediction unit 46 calculates the covariance of the current observation error from the current prediction error. The trajectory prediction unit 46 calculates the Kalman gain from the covariance of the current prediction error and the observation residual. The trajectory prediction unit 46 updates the current state quantity of the effector to a value obtained by adding the estimated value of the current state quantity of the effector to the product of the observation residual and the Kalman gain. The trajectory prediction unit 46 updates the current prediction error to a value obtained by multiplying the prediction error of the residual of the mapping from 1 to the observation space multiplied by the Kalman gain.

The predicted time may be fixed to a predetermined value (for example, 0.2 s to 2 s) or may be variable. The predicted time is set to a value equal to or less than the delay time until the operator's movement is expressed as the movement of the robot 20 (also called the "following delay"). The longer the predicted time, the larger the prediction error of the effector position, which may impair operability.
When the predicted time is variable, a predicted time table showing the predicted time for each operation situation may be stored in advance in the storage unit 44. In particular, if the predicted time exceeds the delay time, the movement of the robot 20 precedes the movement of the operator, which may cause a significant prediction error. Therefore, the delay time may be set as the upper limit of the predicted time. The operation situation estimation unit 34 can estimate the operation situation of the robot 20 using the above method and specify the predicted time corresponding to the estimated operation situation by referring to the predicted time table.

The operation situation estimation unit 34 may set a smaller prediction time for an operation situation requiring accuracy of the position of the effector to be controlled. Such an operation situation is an operation situation in which prediction error of the effector is not tolerated but delay is tolerated. For example, this applies when the object to be operated is a minute object or when the operator operates delicately. The operation situation estimation unit 34 may set a larger prediction time for an operation situation requiring quick response of the effector's position. This operation situation is suitable for an operation situation in which prediction error is tolerated but responsiveness of the effector to operation is expected. For example, this applies when the effector is operated globally toward the target object, when the effector is retreated to a space where no other objects are present, etc.

The operation status estimation unit 34 outputs predicted time information indicating the identified predicted time to the trajectory prediction unit 46. The trajectory prediction unit 46 calculates the position and speed of the effector until the predicted time determined using the predicted time indicated in the predicted time information input from the operation status estimation unit 34. The trajectory prediction unit 46 outputs predicted trajectory information indicating a predicted trajectory indicating at least the position and speed of the effector from the present until the predicted time to the control command generation unit 38.

When generating an operation command to move the effector from the current position toward the target position, the control command generation unit 38 performs known inverse kinematic calculations based on the predicted trajectory input from the trajectory prediction unit 46 and the characteristic parameters input from the operation status estimation unit 34 instead of the drive state information, and determines the angles of each joint so as to move the effector to the convergence position. In contrast, the drive state information up to the present can also be considered as an actual trajectory that indicates the position of the effector at each time up to the present. In a supplementary embodiment, the delay in the operation of the effector can be compensated for by using the predicted trajectory instead of the actual trajectory.

The controller provided in the drive control unit 40 according to this embodiment may calculate the manipulated variable by combining the first component calculated by the feedback term and the second component calculated by the feedforward term. The feedback term is a term that calculates the first component, which is a component of the manipulated variable, based on the deviation between the target value and the output value. The angle of each joint corresponding to the target position is used as the target value. The angle of each joint indicated in the drive state information is used as the output value. This output value corresponds to the current position of the effector at that time. The above PI control and PID control correspond to a method of determining the first component as the manipulated variable without including the second component. In other words, the above proportional term, integral term, and differential term are each included in the first component. The feedforward term is a term that calculates the second component, which is the other component of the manipulated variable, based on the target value without considering the output value. The second component includes a second proportional term that is proportional to the target value. The second proportional term is obtained by multiplying the target value by the second proportional gain.

Therefore, in the drive control parameter table, in addition to the first gain (either the proportional gain, integral gain, or differential gain, or a combination of these) related to the first component, the second gain related to the second component is set as a drive control parameter corresponding to each operating situation. The second gain includes the second proportional gain. The second component may further include one or both of a second integral term proportional to the time integral value of the target value and a second differential term proportional to the time differential value of the target value. The second integral term is obtained by multiplying the target value by the second integral gain. The second differential term is obtained by multiplying the target value by the second differential gain. The second gain further includes one or both of the second integral gain and the second differential gain. And the ratio of the first gain to the second gain may differ depending on the operating situation. For example, the first gain and the second gain may be determined so that the ratio of the first factor to the second factor is relatively large as the operating situation requires greater accuracy in the position of the effector to be controlled. In such operating conditions, the feedback term is relatively emphasized. Also, the first gain and the second gain can be determined so that the ratio of the second factor to the first factor is relatively large in operating conditions that require more immediate response in the position of the effector. In such operating conditions, the feedforward term is relatively emphasized.

The motion status estimation unit 34 refers to the drive control parameter table to identify drive control parameters including a first gain and a second gain corresponding to the estimated motion status, and outputs the identified drive control parameters to the drive control unit 40. The drive control unit 40 calculates a first factor and a second factor for each joint based on the first gain and the second gain indicated in the drive control parameters input from the motion status estimation unit 34, and determines the amount of operation by combining the calculated first factor and second factor. The drive control unit 40 outputs a drive command indicating the amount of operation determined for each joint to the drive unit 24.
In addition, the controller according to the present embodiment may determine the second component as the manipulated variable without including the first component. In that case, the setting of the first gain may be omitted.

(Operation Control Processing)
Next, an example of the operation control process according to the present embodiment will be described. Fig. 7 is a flowchart showing an example of the operation control process according to the present embodiment. The process shown in Fig. 7 further includes steps S122, S124, and S126 in addition to the process shown in Fig. 4.
After the target position estimation unit 36 finishes the process of step S106, the process proceeds to step S122.
(Step S122) The operation status estimation unit 34 refers to the predicted time table and determines a predicted time corresponding to the operation status determined by the operation status estimation unit 34 itself.
(Step S124) The trajectory prediction unit 46 uses the robot movement information and the operation information to estimate a predicted trajectory of the effector from the present to a predicted time that is a set predicted time period.
(Step S126) The control command generator 38 generates a control command indicating the angle of each joint that moves the effector toward the target position as a target value, using the predicted trajectory and the characteristic parameters. Then, the process proceeds to step S110.

In step S122, the operation status estimating unit 34 may refer to the drive control parameter table to identify the drive control parameters corresponding to the operation status.
In step S110, the drive control unit 40 may use the identified drive control parameters to calculate a first factor and a second factor based on the target value and the deviation between the target value and the output value, and determine the manipulated variable by combining the calculated first factor and second factor. The drive control unit 40 generates a drive command indicating the determined manipulated variable, and outputs the generated drive command to the drive unit 24.

As described above, the control device 30 according to the above embodiment includes a trajectory prediction unit 46 that determines a predicted trajectory of the effector of the robot 20 from the current time to a predicted time a predetermined predicted time later, based on at least operation information indicating the operation of the robot 20 and operation information indicating an operation status, and a control command generation unit 38 that generates a control command based on the predicted trajectory.
According to this configuration, the effector of the robot 20 is driven according to a control command generated based on the predicted trajectory of the effector from the current time to a predicted time after the current time, thereby reducing or eliminating the delay (tracking delay) until the operation is reflected in the operation of the robot 20. Since the feeling of operation is improved for the operator, it is possible to achieve both improved work efficiency and reduced burden.

The control device 30 may further include an operation status estimation unit 34 that estimates an operation status of the robot 20 based on at least environmental information indicating an operation environment of the robot 20 and operation information. The trajectory prediction unit 46 may determine a prediction time based on the operation status.
According to this configuration, the predicted time is determined according to the operating condition of the robot 20 estimated from the operating environment and operation condition of the robot 20. Therefore, the balance between the improvement of the operability and the accuracy of the position of the effector to be controlled is adjusted according to the operating condition.

The control device 30 may also include a drive control unit 40 that determines an amount of operation for the movement mechanism based on a target value of the displacement of the movement mechanism of the robot 20 that provides a target position of the effector for each time on the predicted trajectory. The movement status estimator 34 may determine a gain of the target value based on the movement status.
According to this configuration, the contribution of the target value to the amount of operation of the operating mechanism is adjusted in accordance with the operating situation, and therefore the sensitivity of the operation of the effector to the operation by the operator is adjusted in accordance with the operating situation.

The drive control unit 40 may determine the manipulated variable by combining a first component based on the deviation between the output value of the displacement that gives the current position of the effector and the target value and the first gain, and a second component based on the target value and the second gain. The operation status estimating unit 34 may determine the first gain and the second gain based on the operation status.
According to this configuration, the balance between the feedback term and the feedforward term is adjusted according to the operating situation, and therefore the balance between the sensitivity and accuracy of the operation of the effector organ in response to the operation of the operator is adjusted according to the operating situation.

Furthermore, the motion status estimating unit 34 may further estimate the motion status based on operator information indicating the status of the operator operating the robot.
According to this configuration, the operating status can be accurately estimated by further referring to the status of the operator, which further improves the work efficiency and reduces the workload of the robot.

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to the above, and various design changes can be made without departing from the spirit of the present invention.

The above-described embodiments and their modified examples may be combined with each other as long as no contradiction occurs, or some of them may be omitted.
For example, in each of the above embodiments, the target position estimation unit 36 may be omitted, and further, the processing based on the target position estimated by the target position estimation unit 36 may be omitted.
In the second embodiment, when the predicted time is fixed, the operation status estimating unit 34 may be omitted, and further, the processing based on the operation status may be omitted.

In the environmental information acquisition unit 80, the distance measurement unit 84 may be integrated with the image capture unit 82 and configured as a three-dimensional image capture unit. The environmental information acquisition unit 80 may also include a multi-viewpoint capture unit instead of the image capture unit 82 and the distance measurement unit 84. The multi-viewpoint capture unit is an image capture unit that has multiple viewpoints and can capture images for each viewpoint. The multi-viewpoint capture unit includes a so-called stereo camera. The operating environment of the robot 20 is represented three-dimensionally by the images for each viewpoint (multi-viewpoint images) captured by the multi-viewpoint capture unit. The target position estimation unit 36 can identify the area occupied by an object from the operating environment of the robot 20 by analyzing the multi-viewpoint images.

Also, a part or the whole of the display device 50 may be omitted. Although the gaze information mainly indicates the gaze direction of one of the operator's eyes (e.g., the left eye) in the above description, the present invention is not limited to this. The gaze information may indicate the gaze direction of each of the left eye and the right eye. When the gaze direction of one eye is unavailable at a certain time (e.g., blinking), the target position estimation unit 36 may use the gaze direction of the other eye when determining the point of attention. When the gaze directions of both eyes are available, the target position estimation unit 36 may determine the intersection of the line segments of the gaze directions of each of the two eyes, or the midpoint of the closest point of each of these line segments, as the gaze point. Then, the target position estimation unit 36 may determine the direction from the head to the gaze point as the gaze direction representing the gaze directions of both eyes, and determine the intersection of the determined gaze direction and the surface of the object. The determined intersection is used to set the target position.

The robot 20 may be equipped with multiple manipulators. The multiple manipulators may correspond to one forearm and one hand of each of multiple operators. Each manipulator can be operated in response to the movement of one forearm and one hand of each of the operators. The control device 30 may determine one target position for each manipulator using the above method and operate the manipulator toward the determined target position. The robot 20 may be a dual-arm robot equipped with two manipulators. For a dual-arm robot, for example, one and the other manipulator may correspond to the left and right hands of one operator, respectively.

S1...Robot system, 20...Robot, 24...Drive unit, 26...Drive sensor, 28...Power source, 30...Control device, 32...Information acquisition unit, 34...Operation status estimation unit, 36...Target position estimation unit, 38...Control command generation unit, 40...Drive control unit, 42...Communication unit, 44...Memory unit, 46...Trajectory prediction unit, 50...Display device, 52...Display unit, 54...Gaze detection unit, 56...Movement detection unit, 58...Control unit, 60...Communication unit, 70...Operation device, 72...Movement detection unit, 74...Control unit, 76...Communication unit, 80...Environment information acquisition unit, 82...Photographing unit, 84...Distance measurement unit, 86...Communication unit

Claims (7)

a trajectory prediction unit that determines a predicted trajectory of the effector of the robot from a current time to a predicted time after a predetermined predicted time based on motion information indicating at least a motion of the robot and operation information indicating at least a hand position of an operator;
a control command generating unit that generates a control command based on the predicted trajectory;
a motion status estimation unit that estimates a motion status of the robot based on at least environmental information indicating an operational environment of the robot and the operation information;
The trajectory prediction unit is
The predicted time is determined based on the operating condition.
Control device. a drive control unit that determines an amount of operation for the movement mechanism of the robot based on a target value of a displacement of the movement mechanism of the robot that provides a target position of the effector for each time on the predicted trajectory,
The operation status estimation unit is
The control device according to claim 1 , wherein the gain of the target value is determined based on the operating condition. The drive control unit is
the operation status estimation unit determines the manipulated variable by combining a first component based on a deviation between the output value of the displacement that provides a current position of the effector and the target value and a first gain, and a second component based on the target value and a second gain;
The control device according to claim 2 , wherein the first gain and the second gain are determined based on the operating condition. The operation status estimation unit is
The control device according to claim 1 , further comprising: a controller configured to estimate the operation status based on operator information indicating a status of an operator operating the robot. A program for causing a computer to function as the control device according to any one of claims 1 to 3 . A robot system comprising the control device according to any one of claims 1 to 3 and the robot. A control method in a control device, comprising:
a first step of determining a predicted trajectory of an effector of the robot from a current time to a predicted time after a predetermined predicted time based on motion information indicating at least a motion of the robot and operation information indicating at least a hand position of an operator;
a second step of generating control commands based on the predicted trajectory;
and a third step of estimating an operating condition of the robot based on environmental information indicating at least an operating environment of the robot and the operation information,
The first step comprises:
determining the predicted time based on the operating condition;
Control methods.

Images