JP7703904B2 - Servo system and processing method - Google Patents

Description

The present invention relates to a processing device, a servo system, a processing method, and a program.

There is a technology called visual servoing that controls the movement of a robot based on visual information (images) acquired by a sensor. For example, by executing visual servoing, a robot can perform operations such as placing an object grasped by an end effector on a target object.

Patent document 1 describes a technology that calculates the trajectory of a robot when performing visual servoing based on the features of an image captured by a camera (sensor).

U.S. Pat. No. 8,774,967

On the other hand, in visual servoing, the grasped object or target object may be cut off or may overlap in the image acquired by the sensor. In this regard, Patent Document 1 only discloses calculating a robot trajectory that avoids prohibited movement areas in three-dimensional space, but does not take into consideration the grasped object or target object being cut off or overlapping in the image acquired by the sensor. For this reason, in the past, there were cases where only a small portion of the grasped object or target object was included in the image acquired by the sensor, resulting in a problem of reduced accuracy and/or speed of visual servoing.

The present invention aims to provide a technique for improving the accuracy or speed of visual servoing.

To achieve the above objective, the present invention adopts the following configuration.

That is, a processing device according to one aspect of the present invention is a processing device that, in a servo system that performs visual servoing, has a sensor that captures and detects an object and a robot equipped with an end effector, and determines a servo trajectory, which is at least one trajectory related to the servo system, and a starting posture of the robot at the start of the visual servoing, and has an acquisition means that acquires information on the initial posture of the robot and information on the target posture of the robot at the end of the visual servoing, a planning means that 1) determines multiple candidates for the start posture, and 2) generates multiple trajectories of the end effector when the posture of the robot changes from the initial posture to the target posture via one of the multiple candidates, and a determination means that determines, from the multiple candidates for the servo trajectory corresponding to each of the multiple trajectories of the end effector, the candidate with the highest evaluation value for the accuracy or speed of the visual servoing as the servo trajectory, and determines the candidate for the start posture corresponding to the servo trajectory as the start posture.

According to such a configuration, the starting posture of the robot at the start of visual servoing can be determined without the user having to manually set it. In other words, the burden on the user in visual servoing can be reduced. Furthermore, the control trajectory with the highest evaluation value is selected as the servo trajectory from among multiple candidate servo trajectories. Therefore, the servo trajectory can be determined to be a trajectory along which the robot can easily move and/or a trajectory along which the sensor moves to a position where the target object can be easily grasped. Therefore, for example, the probability of failure of visual servoing, which performs an operation of connecting a grasped object to a target object, can be reduced. In other words, the accuracy of visual servoing can be improved. Furthermore, the time required for visual servoing can be reduced.

In the above processing device, the planning means may determine a plurality of candidates for the start posture from among postures between the initial posture and the target posture. A posture between the initial posture and the target posture is, for example, a posture that will be passed through in the change when the planning means generates a change in the posture of the robot according to a predetermined algorithm so that the posture of the robot changes from the initial posture to the target posture. With this configuration, the change from the initial posture to the target posture that passes through the start posture can be determined to be a more efficient change, so that the servo trajectory can be set to a trajectory that is easier for the robot to achieve more efficiently.

The processing device may include at least one of the trajectory of the sensor, the trajectory of the relative position change of the object with respect to the end effector, and the trajectory of the object in the captured image captured by the sensor.

The above processing device may further include a simulation means for generating a plurality of candidates for the servo trajectory corresponding to each of a plurality of trajectories of the end effector, and simulating visual servoing according to each of the plurality of candidates for the servo trajectory, and the simulation means may correct each of the plurality of candidates for the trajectory of the servo system based on the simulation result, and the determination means may determine the servo trajectory from among the plurality of corrected candidates for the servo trajectory. With this configuration, the servo trajectory can be determined based on the result of the visual servo simulation, so that it is possible to determine a servo trajectory corresponding to a trajectory along which the end effector can realistically move. In other words, it is possible to determine a servo trajectory that satisfies the operational constraints imposed on the end effector.

In the above processing device, the servo system has a first controller and a second controller, the first controller determines the position and orientation of the end effector corresponding to each of the multiple trajectories of the end effector, the second controller controls the posture of the robot to realize the position and orientation of the end effector determined by the first controller, and the simulation means has a third controller that emulates the first controller and a fourth controller that emulates the second controller, and the visual servo may be simulated using the third controller and the fourth controller. With this configuration, a servo trajectory can be determined based on the results of a simulation according to the second controller that controls the posture of the robot, so that a servo trajectory corresponding to a trajectory on which the robot can realistically move can be determined. In other words, a servo trajectory that satisfies the operating constraints imposed on the robot can be determined.

The processing device may further include an evaluation means for evaluating each of the multiple candidates for the servo trajectory.

In the above processing device, the evaluation means may calculate an evaluation value for each of the multiple candidates for the servo trajectory based on a task score for the movement of the robot and a sensing score for the detection of the target object when the visual servo is executed according to each of the multiple candidates for the servo trajectory. With this configuration, it is possible to determine a servo trajectory that allows the robot to move easily and detects (images) the entire target object.

In the above processing device, the task score may be a score based on at least one of the time required for the robot to move and the difference between the robot's posture at the end of the visual servoing and the target posture. If the task score is based on the time required for the robot to move, a servo trajectory that allows the robot to move faster can be determined. If the task score is the difference between the robot's posture at the end of the visual servoing and the target posture, a servo trajectory that can change the robot's posture closer to the target posture can be determined.

In the above processing device, the servo system may be capable of detecting the object, and the sensing score may be a score based on at least one of the total number of expected invalid detections during execution of visual servoing, the distance between the object and a detectable range in which the servo system can detect an object, and the total number of expected detection errors. With this configuration, it is possible to determine a servo trajectory that can detect (image) the object more accurately.

In the above processing device, the expected total number of invalid detections may be the number of times that the sensor performs imaging and at least one of the following occurs: the field of view or the detectable range of the sensor does not include a first percentage or more of the object, and another object overlaps an area of the object that is a second percentage or more of the entire area in the captured image captured by the sensor. With this configuration, it is possible to reduce the number of occurrences in which the object is located in a place that cannot be detected (captured) by the sensor in the captured image, or the object is overlapped by another object by a predetermined percentage or more (occlusion occurs), thereby preventing the range in which the object can be detected from becoming narrower.

In the above processing device, the planning means may determine a plurality of candidates for the starting posture from a plurality of postures of the robot corresponding to a plurality of positions and postures of the end effector located within a predetermined range. With this configuration, for example, the user can simply specify a predetermined range of any size (e.g., a spherical range) and the processing device can determine the starting posture. This reduces the processing burden on the user and allows the processing device to determine an appropriate starting posture.

In the above processing device, the planning means may select a plurality of candidates for the starting posture from a plurality of postures of the robot corresponding to a plurality of positions and postures of the sensor where no other object is placed between the sensor and the object when the sensor captures an image and where the object is included in the field of view of the sensor. With this configuration, it is possible to determine, as the starting posture, a posture of the robot corresponding to a position and posture of the sensor where no occlusion occurs and where the entire object can be captured. Therefore, at the start of visual servoing, the sensor can be placed in a position and posture where the object can be detected (captured) more accurately.

The present invention may also be a servo system comprising the above-mentioned processing device, a robot having the end effector, a sensor for capturing an image of the object, and a controller for operating the robot according to the servo trajectory determined by the processing device and for starting the visual servo when the robot changes to the starting posture. With this configuration, visual servoing can be performed using a robot and a sensor based on the appropriate servo trajectory determined by the processing device, as described above.

The present invention may be understood as an apparatus having at least a part of the above means, or as an electronic device, a control system, an information processing system, an information processing device, a servo device, or a system. The present invention may also be understood as a control method, a processing method, a servo method, or a generating method including at least a part of the above processing. The present invention may also be understood as a program for implementing such a method, or a recording medium (storage medium) on which the program is non-temporarily recorded. Note that the above means and processing can be combined with each other as much as possible to constitute the present invention.

The present invention improves the accuracy or speed of visual servoing.

FIG. 1 is an external view showing a servo system. FIG. 2A is a diagram illustrating a captured image, FIG. 2B is a diagram illustrating a detectable range, and FIG. 2C is a diagram illustrating a candidate range. FIG. 3 is a diagram showing the internal configuration of the control device and the processing device. FIG. 4 is a flowchart showing the process of determining the optimum trajectory.

Below, an embodiment of the present invention will be described with reference to the drawings.

<Application Examples>
In the following, a technique for determining the starting posture of the robot 10 at the start of visual servoing and the proper trajectory (servo trajectory; trajectory of the sensor 20, the object 2, etc.) of the servo system 1 when visual servoing is performed will be described before visual servoing is performed in the servo system 1 that performs visual servoing. Specifically, the servo system 1 generates a plurality of candidates for the starting posture. The servo system 1 also generates candidates for the proper trajectory when the robot 10 is moved so as to change from the initial posture to a preset target posture (the posture of the robot 10 that is the target at the end of visual servoing) via one of the plurality of candidates for the starting posture. Then, the servo system 1 selects one proper trajectory from the plurality of candidates for the proper trajectory based on the evaluation values of these proper trajectories. As a result, one proper trajectory is selected, and the starting posture corresponding to the proper trajectory is selected. Here, the evaluation value is an evaluation value related to the accuracy or speed of the visual servoing. The evaluation value is a value based on the task score related to the movement of the robot 10 and the sensing score related to the detection of the object 2.

This allows the user to determine the starting posture of the robot 10 at the start of visual servoing without having to manually set it. In other words, the burden on the user in visual servoing can be reduced. Furthermore, the control trajectory with the highest evaluation value is selected as the appropriate trajectory from among multiple suitable trajectory candidates. Therefore, the appropriate trajectory can be determined to be a trajectory that is easy for the robot 10 to move, and/or a trajectory that moves the sensor 20 to a position where the target object 2 can be easily grasped. Therefore, for example, the probability of failure of visual servoing, which performs an operation to connect the grasped object 3 to the target object 2, can be reduced. In other words, the accuracy of visual servoing can be improved. And the time required for visual servoing can be reduced.

<Embodiment 1>
A servo system 1 according to the first embodiment will be described below. The servo system 1 executes visual servoing (operation of the robot 10 based on visual information). In the first embodiment, the servo system 1 controls the posture of the robot 10 so that the grasped object 3 grasped by the robot 10 is connected to an object 2 (target; a specified object) while capturing images of the grasped object 3 and the object 2. Note that the term "trajectory" in the following description may be a line segment showing the trajectory along which the object moves, or may be information showing multiple positions and the order of movement on the line segment. In the present embodiment, the trajectory of a certain object indicates the position and posture of the object when it moves.

[Servo system configuration]
First, the configuration of a servo system 1 will be described with reference to the configuration diagram of Fig. 1. The servo system 1 has a robot 10, a sensor 20, a control device 30, and a processing device 40. The servo system 1 can be regarded as having an object 2 and a grasped object 3, or as not having the object 2 and the grasped object 3.

The robot 10 controls the position and orientation of the sensor 20 and the end effector 11 by changing its own orientation. In this embodiment, the orientation of the robot 10 indicates the angle of the joint of the arm of the robot 10 and the rotation angle of the end effector 11. Therefore, in this embodiment, the orientation of the robot 10 and the position and orientation of the end effector 11 correspond one-to-one. The orientation of the robot 10 is controlled by the control device 30. The sensor 20 is provided at the tip of the arm of the robot 10. Therefore, the robot 10 may be considered to have the sensor 20. Note that, although the "position and orientation" refers to the position and orientation in this embodiment, it may be the position or the orientation as long as no technical contradiction occurs. Also, the "position and orientation" can be regarded as six-dimensional information of a certain object (information representing the three-dimensional coordinates of a certain object and the three axial directions of the object).

The robot 10 has an end effector 11 (gripping part) and a base 12. The end effector 11 grasps the grasped object 3. The base 12 fixes the position of one end of the robot 10. In the following, the posture of the robot 10 at the start of visual servoing is called the "start posture". Also, the posture of the robot 10 specified by the user and which is the target of visual servoing is called the "target posture". This target posture is a posture arbitrarily set by the user, so it does not have to be a posture in which the grasped object 3 is connected to the target object 2.

The sensor 20 captures an image of the target object 2 and the object to be grasped 3. In this embodiment, the relative positional relationship between the sensor 20 and the object to be grasped 3 is fixed. Therefore, the sensor 20 includes the object to be grasped 3 in its field of view (FOV; Field Of View), which is the range in which the sensor 20 can capture images, as long as no other object overlaps the object to be grasped 3. The image captured by the sensor 20 includes the target object 2 and the object to be grasped 3, for example, as shown in FIG. 2A.

The control device 30 acquires the captured image from the sensor 20 and acquires information on the current posture of the robot 10 (such as information on the motor rotation angle) from the robot 10. As a result, the control device 30 detects the target object 2 and the grasped object 3 (their position and posture) located in the detectable range 60 of the visual field based on the captured image and the posture of the robot 10 corresponding to the captured image. The detectable range 60 is a range that depends not only on the visual field of the sensor 20 but also on the distance from the sensor 20 as shown in FIG. 2B. For example, the control device 30 can detect the position and posture of the target object 2 and the grasped object 3 based on the difference between the captured images captured by the sensor 20 from two different positions and postures and the posture of the robot 10 corresponding to the two different positions and postures. In addition, the control device 30 controls the posture of the robot 10 to control the positions and postures of the end effector 11 and the sensor 20. As a result, the control device 30 controls the robot 10 so as to move the grasped object 3 to a desired position in the captured image (performs visual servoing).

The processing device 40 determines the starting posture of the robot 10 when the servo system 1 executes visual servoing (movement of the robot 10 based on visual information). The processing device 40 also determines a proper trajectory (servo trajectory) of the servo system 1, which is a set of the trajectory of the sensor 20 (sensor trajectory), a trajectory (object trajectory) indicating a relative position change between the object 2 and the end effector 11, and a trajectory (visual trajectory) of a visual feature of the object 2 in a captured image. Here, the visual feature of the object 2 is, for example, one edge, one corner, or the center position of the object 2. The servo system 1 starts visual servoing from the starting posture of the robot 10, and continues to execute visual servoing while controlling the robot 10 according to the proper trajectory.

[Configuration of the control device and processing device]
3, detailed configurations of the control device 30 and the processing device 40 will be described. The control device 30 has a simulator 310 and a controller 320. The processing device 40 has an input/output unit 410, a planning unit 420, a simulation unit 430, and an evaluation unit 440.

The simulator 310 detects (estimates) the target object 2 and the grasped object 3 (the position and orientation of the target object 2, etc.) based on the captured image acquired from the sensor 20 and the orientation of the robot 10.

The controller 320 includes a visual controller that controls the position and orientation of the end effector 41, and a robot controller that controls the position and orientation of the robot 10. The visual controller determines the orientation of the end effector 11 corresponding to the trajectory of the end effector 11. The robot controller outputs a command to drive the motor of the robot 10 in order to realize the change in the orientation of the end effector 11 determined by the visual controller.

The controller 320 generates (outputs) a command to control the posture of the robot 10 based on the appropriate trajectory acquired from the processing device 40. The controller 320 then outputs a command to the sensor 20 to cause the sensor 20 to capture images during the period in which the posture of the robot 10 is changing from the starting posture. Through this processing, the controller 320 can execute visual servoing when the posture of the robot 10 changes to the starting posture while moving the robot 10 based on the appropriate trajectory.

If the target object 2 moves, or if the target object 2 and the grasped object 3 cannot be connected in the target position and orientation set by the user (if the visual servo goal is not achieved), the operation of the robot 10 needs to be corrected. In this case, the controller 320 may generate a command to control the robot 10 so as to correct the operation of the robot 10 based on the position and orientation of the target object 2, the position of the target object 2 in the captured image, etc.

The input/output unit 410 is a user interface that accepts (acquires) user operations (user inputs). The user can input (set) the initial posture and target posture of the robot 10 via the input/output unit 410. The user can also input (set) a candidate range 50 (a predetermined range in a three-dimensional space) that is a range of candidates for the position and posture of the end effector 11 when starting the execution of visual servoing, as shown in FIG. 2C, via the input/output unit 410. Furthermore, the user can input information such as parameters of the controller 320 and the virtual controller 431, three-dimensional models of the target object 2 and the grasped object 3, and operation constraints of the robot 10 via the input/output unit 410. The user can also input information of an area (prohibited movement area) in which the robot 10 is prohibited from entering via the input/output unit 410. In this embodiment, the candidate range 50 is a spherical area, but may be an area of any shape. The planning unit 420 may also determine, as the candidate range, a sphere of a predetermined radius whose center point is the center between the position of the end effector 11 corresponding to the initial posture of the robot 10 and the position of the end effector 11 corresponding to the target posture of the robot 10.

In addition, the user may input, via the input/output unit 410, information on the movement of the object 2 (for example, the object 2 is assumed to be placed on a conveyor rotating at a constant speed).
It is also possible to input movement information of the object 2 to the .

Furthermore, the input/output unit 410 outputs information on the appropriate trajectories (sensor trajectory, object trajectory, and visual trajectory) determined by the processing device 40 and information on the starting posture to the control device 30.

The planning unit 420 determines, as the multiple candidate postures, multiple candidates for the starting posture such that the end effector 11 is located within the candidate range 50. Note that the planning unit 420 may determine, as the candidate postures, postures of the robot 10 such that the target object 2 and the grasped object 3 do not overlap in the captured image captured by the sensor 20 (no other object exists between the sensor 20 and the target object 2). Alternatively, the planning unit 420 may determine the candidate postures such that a part of the target object 2 and the grasped object 3 is not located outside the field of view range or/and the detectable range 60. In other words, the planning unit 420 may select multiple candidates for the starting posture from multiple postures of the robot 10 corresponding to each of the multiple positions and postures of the sensor 20 in which the target object 2 is included in the field of view range of the sensor 20.

Then, the planning unit 420 generates multiple candidate trajectories (effector trajectories) of the end effector 11 when the posture of the robot 10 changes from the initial posture to the target posture via one of the multiple candidate postures. For example, if the planning unit 420 generates 10 candidate postures, it repeats the generation of candidate trajectories of the end effector 11 when the posture of the robot 10 changes from the initial posture to the target posture via one candidate posture 10 times. Note that any path calculation algorithm can be used to calculate the effector trajectory.

The simulation unit 430 generates visual trajectory candidates, object trajectory candidates, and sensor trajectory candidates (these three types of trajectory candidates are referred to as a set of "non-controlled trajectories") corresponding to each of the multiple effector trajectory candidates. The simulation unit 430 then calculates (simulates) the movement of the robot 10 such that the object 2 passes through multiple points in each visual trajectory candidate. The simulation unit 430 then corrects the visual trajectory candidates based on the simulation results. For example, when the object 2 cannot pass through multiple points in the visual trajectory candidate due to the movement restrictions of the robot 10, the simulation unit 430 corrects the visual trajectory candidate by correcting (shifting) the positions of the multiple points. The simulation unit 430 also performs such correction on the object trajectory candidates and the sensor trajectory candidates. In the following, the corrected non-controlled trajectory (one set of visual trajectory candidates, object trajectory candidates, and sensor trajectory candidates; proper trajectory candidates) is referred to as a "controlled trajectory".

The simulation unit 430 also has a virtual controller 431. The virtual controller 431 is software (an emulator of the controller 320) that realizes the same processing as the controller 320, and is set with the same parameters as the controller 320. For this reason, the virtual controller 431 has a virtual visual controller that emulates (mimics) the visual controller of the controller 320, and a virtual robot controller that emulates (mimics) the visual controller of the controller 320. The virtual visual controller determines the change in the posture position of the end effector 41 corresponding to each of the multiple effector trajectories. The virtual robot controller controls the posture of the robot 10 to realize the change in the posture position of the end effector 41 determined by the virtual visual controller.

The simulation unit 430 uses a virtual visual controller and a virtual robot controller to simulate the movement of the robot 10 such that the target object 2 passes through a plurality of points on each visual trajectory. By using the virtual visual controller and the virtual robot controller in this manner, movements that the robot 10 cannot realize can be eliminated.

The evaluation unit 440 evaluates each of the multiple control trajectories generated by the simulation unit 430. Specifically, the evaluation unit 440 calculates an evaluation value for each of the multiple control trajectories based on a task score related to the movement of the robot 10 and a sensing score related to the detection of the object 2. Therefore, the evaluation value is a value related to the accuracy or speed of visual servoing. For example, the evaluation value is larger as the task score is smaller, and the evaluation value is larger as the sensing score is smaller. Note that the task score may be any score as long as the control trajectory that makes it easier for the robot 10 to move is a lower value, or the control trajectory that can change the posture of the robot 10 closer to the target posture is a lower value. Also, the sensing score may be any score as long as the detection accuracy of the object 2 (the degree of match between the actual object 2 and the detection result) is higher.

The task score is, for example, a value based on at least one of the error (final posture error) between the final posture of the robot 10 in a simulated case and the target position and posture, and the total time (total movement time) for the robot 10 to move according to the control trajectory. The task score is, for example, a weighted sum of the final posture error and the total movement time, or a multiplication of the final posture error and the total movement time.

The sensing score is, for example, a value based on at least one of the total number of expected invalid detections (total invalid detections), the distance between the detectable range 60 and the object 2 (object distance), and the total number of expected detection errors (total detection errors) while visual servoing is being performed. The object distance can be the average, mode, median, minimum, or maximum value of the distance between the center or outer periphery of the detectable range 60 and the center or outer periphery of the object 2 while visual servoing is being performed. The sensing score is, for example, a value obtained by adding the total invalid detections, the object distance, and the total detection errors with a weight, or a value obtained by multiplying the total invalid detections, the object distance, and the total detection errors.

The total number of invalid detections is, for example, the number of times that the sensor 20 has performed imaging, in which either the object 2 is not included in the detectable range 60 or the field of view range, or another object is placed between the sensor 20 and the object 2 (occlusion has occurred). The total number of invalid detections may also be the number of times that the sensor 20 has performed imaging, in which either the object 2 is not included in the detectable range 60 or the field of view range at a first ratio or more, or another object overlaps with an area of the object 2 that is a second ratio or more of the entire area of the object 2 in the captured image captured by the sensor 20. For example, in a case where 2 cm ² of the object 2 is captured in the captured image if a certain object does not overlap with the object 2, if a certain object overlaps with the object 2 by 1.2 cm ² or more, which is 60% (second ratio) of 2 cm ² in the captured image, it can be said that an invalid detection has occurred. Note that, instead of the total number of invalid detections, the reciprocal of the average value of the ratio of the area included in the detectable range 60 or the field of view range to the entire area of the object 2 may be used.

The detection error is, for example, a value based on the variance of visual features corresponding to images captured at two points in time. Specifically, the detection error is a value based on the variance of the positions of visual features of images obtained by projecting (projecting) the two captured images onto the surface of the object 2. However, without being limited to this, the detection error may be any value that increases as the detection accuracy of the sensor 20 (degree of match between the actual object and the detected object) decreases.

Further, the control device 30 and the processing device 40 can be configured by a computer including, for example, a CPU (processor), a memory, a storage, etc. In this case, the configuration shown in FIG. 3 is realized by loading a program stored in the storage into the memory and executing the program by the CPU. Such a computer may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smartphone, or may be an embedded computer such as an on-board computer. Alternatively, all or a part of the configuration shown in FIG. 3 may be configured by an ASIC, an FPGA, or the like. Alternatively, all or a part of the configuration shown in FIG. 3 may be realized by cloud computing or distributed computing.

[Process for determining appropriate trajectory]
Next, the process (method) of determining the optimum trajectory executed by the processing device 40 will be described with reference to the flowchart of FIG.

In step S1001, the input/output unit 410 acquires user-defined information. The user-defined information includes information on the three-dimensional model (models of the robot 10, the sensor 20, the target object 2, the grasped object 3, and other external environment), information on the parameters of the controller 320 (visual controller and robot controller), and information on the size and shape of the candidate range 50. The user-defined information also includes, for example, information indicating the method of calculating the task score and sensing score described above, information on the motion constraints (restrictions on the robot 10, the sensor 20, and other external environment), and information on the generator (software) used when generating the initial posture and effector trajectory.

The operational constraints of the robot 10 are conditions such as the operable angle of each joint of the robot 10, the relationship between the angle of each joint and the end effector 11, the speed and acceleration of change of each joint, the movable range of the end effector 11, and the torque of the motor of the robot 10. The operational constraints of the sensor 20 are the conditions of the field of view (FOV) of the sensor 20 and the detectable range 60.

In step S1002, the planning unit 420 uses a generator for generating the starting posture to determine multiple candidates for the starting posture such that the end effector 11 is located within the candidate range 50 as multiple candidate postures. The planning unit 420 may further determine multiple candidate postures such that the end effector 11 is located within the candidate range 50 from among postures between the initial posture and the target posture of the robot 10. In addition, the planning unit 420 determines, for example, a posture of the robot 10 such that the target object 2 and the grasped object 3 do not overlap in the captured image captured by the sensor 20 (for example, such that no other object is placed between the sensor 20 and the target object 2) as a candidate posture based on information of a three-dimensional model. Alternatively, the planning unit 420 determines, as a candidate posture, a posture of the robot 10 such that all of the target object 2 and the grasped object 3 are located within the field of view or the detectable range 60. Furthermore, the planning unit 420 may limit the candidate postures to postures to which the robot 10 can change, or to postures to which the robot 10 is not positioned in a prohibited movement area, based on the information on the motion constraints.

In step S1003, the planning unit 420 generates multiple effector trajectory candidates using a generator used for generating the effector trajectory. Here, each of the multiple effector trajectory candidates is generated so as to correspond to the change in the posture of the robot 10 from the initial posture to the target posture via one of the multiple candidate postures. In addition, the planning unit 420 generates multiple effector trajectory candidates, for example, based on information on the motion constraints so that the robot 10 does not move to a prohibited movement area. Note that, before generating the multiple candidate postures, the planning unit 420 may generate multiple effect trajectories corresponding to the change in the posture of the robot 10 from the initial posture to the target posture. Then, the planning unit 420 may determine, as multiple candidate start postures (multiple candidate postures), postures of the robot 10 in which the end effector 11 is located inside the candidate range 50, from among the multiple effect trajectories.

In step S1004, the simulation unit 430 generates a plurality of uncontrolled trajectories corresponding to the plurality of effector trajectories, respectively, based on the information on the three-dimensional model and the limiting conditions. Then, the simulation unit 430 uses the virtual controller 431 to simulate the movement of the robot 10 in which the sensor 20 and the object 2 pass through a plurality of points (a plurality of positions and orientations) on each uncontrolled trajectory.

In step S1005, the simulation unit 430 generates a plurality of control trajectories. Specifically, for each of the plurality of non-control trajectories, the simulation unit 430 corrects the non-control trajectory so as to correspond to the movement of the robot 10 simulated in S1004, and generates a control trajectory (a candidate for a suitable trajectory) corresponding to the non-control trajectory.

In step S1006, the evaluation unit 440 evaluates each of the multiple control trajectories. Specifically, the evaluation unit 440 calculates an evaluation value for each of the multiple control trajectories based on the task score and sensing score described above.

In step S1007, the planning unit 420 selects the one with the highest evaluation value from among the multiple control trajectories as the appropriate trajectory. For example, the planning unit 420 may select the one with the highest evaluation value from among the control trajectories whose task scores are smaller than a first threshold value and whose sensing scores are smaller than a second threshold value as the appropriate trajectory. This ensures that the task scores and sensing scores are smaller than predetermined values. The planning unit 420 also selects a candidate posture corresponding to the appropriate trajectory (the posture of the robot 10 at the start of visual servoing when the servo system 1 is executed according to the appropriate trajectory) as the starting posture.

In this way, the processing device 40 generates (determines) the appropriate trajectory and starting posture. This allows the user to determine the starting posture of the robot 10 at the start of visual servoing without having to set it manually. In other words, the burden on the user in visual servoing can be reduced. The processing device 40 can then set the starting posture to a posture in which no occlusion occurs or a posture in which the target object 2 is located within the field of view or detectable range 60. In other words, the processing device 40 can determine an optimal starting posture for performing visual servoing.

Furthermore, the processing device 40 selects the control trajectory with the highest evaluation value from among the multiple control trajectories as the appropriate trajectory. The evaluation value is a value related to the accuracy and/or speed of visual servoing. Therefore, the appropriate trajectory can be a trajectory along which the robot 10 can easily move, and/or a trajectory along which the sensor 20 moves to a position that makes it easy to grasp the target object 2. In other words, the probability of failure of visual servoing, which performs the operation of connecting the grasped object 3 to the target object 2, can be reduced (the accuracy of visual servoing is improved). Furthermore, the time required for visual servoing can be reduced.

In the above description, the servo system 1 detects the position and orientation of the target object 2 and the grasped object 3 in the control device 30 based on captured images between two points in time, but this is not limited to the above. For example, the sensor 20 may be a three-dimensional sensor, and the sensor 20 itself may detect the position and orientation of the target object 2 and the grasped object 3.

In this embodiment, the effector trajectory and the sensor trajectory are the trajectories of the end effector 11 and the sensor 20 when the posture of the robot 10 changes from the initial posture to the target posture via the start posture.
It may be the trajectory of the end effector 11 or the sensor 20 when the posture of zero changes from the start posture to the target posture. This is because visual servoing is not executed during the period until the robot 10 changes from the initial posture to the start posture.

In addition, in the proper trajectory, the controlled trajectory, and the uncontrolled trajectory, the effector trajectory corresponding to the sensor trajectory may be used instead of the sensor trajectory. This is because in this embodiment, the relative positional relationship between the sensor 20 and the end effector 11 is fixed. Note that the change in posture of the robot 10 may be used instead of the sensor trajectory.

Furthermore, if the object 2 does not move during visual servoing, the object trajectory is not necessary for the proper trajectory, the controlled trajectory, and the uncontrolled trajectory. This is because the positional relationship between the end effector 11 and the object 2 can be calculated from the sensor trajectory.

When performing visual servoing to change the grasped object 3 to a desired position and orientation in the two-dimensional space of the captured image, the visual trajectory is required for the proper trajectory, the controlled trajectory, and the non-controlled trajectory, as described above. On the other hand, when performing visual servoing to change the grasped object 3 to a desired position and orientation in three-dimensional space, the visual trajectory is not necessarily required for the proper trajectory, the controlled trajectory, and the non-controlled trajectory. In other words, as long as the proper trajectory (servo trajectory), the controlled trajectory, and the non-controlled trajectory include the trajectory of at least one component (object; member) related to the servo system 1, it is possible to obtain the same effect as in this embodiment.

Note that the interpretation of the claims is not limited solely to the matters described in the embodiments. The interpretation of the claims also includes the scope of the description that is sufficient to enable a person skilled in the art to recognize that the problem to be solved by the invention can be solved, taking into account the common general technical knowledge at the time of filing.

(Appendix 1)
A servo system (1) for performing visual servoing, comprising: a sensor (20) for capturing and detecting an object (2); and a robot (10) having an end effector (11),
a processing device (40) for determining at least one trajectory, a servo trajectory, for the servo system (1) and a starting posture of the robot (10) at the start of the visual servoing,
An acquisition means (410) for acquiring information on an initial posture of the robot (10) and information on a target posture of the robot (10) at the end of the visual servoing;
a planning means (420) for 1) determining a plurality of candidates for the start posture, and 2) generating a plurality of trajectories for the end effector (11) when the posture of the robot (10) changes from the initial posture to the target posture via any one of the plurality of candidates;
a determination means (420) for determining, as the servo trajectory, a candidate having the highest evaluation value regarding the accuracy or speed of the visual servo from among a plurality of candidates of the servo trajectory corresponding to each of a plurality of trajectories of the end effector (11), and determining, as the starting posture, a candidate of a starting posture corresponding to the servo trajectory;
A processing device (40) comprising:

(Appendix 2)
A servo system (1) that performs visual servoing and has a sensor (20) that captures and detects an object (2) and a robot (10) equipped with an end effector (11),
A processing method for determining at least one trajectory, a servo trajectory, for the servo system (1) and a starting posture of the robot (10) at the start of the visual servoing, comprising:
An acquisition step (S1001) of acquiring information on an initial posture of the robot (10) and information on a target posture of the robot (10) at the end of the visual servoing;
1) determining a plurality of candidates for the starting posture, and 2) generating a plurality of trajectories of the end effector (11) when the posture of the robot (10) changes from the initial posture to the target posture via any one of the plurality of candidates (S1002, S1003);
a determining step (S1007) of determining, as the servo trajectory, a candidate having the highest evaluation value regarding the accuracy or speed of the visual servo from among a plurality of candidates of the servo trajectory corresponding to each of a plurality of trajectories of the end effector (11), and determining, as the starting posture, a candidate of a starting posture corresponding to the servo trajectory;
A processing method comprising the steps of:

1: servo system, 2: object, 3: grasped object,
10: robot, 11: end effector, 12: base,
20: sensor, 30: control device, 40: processing device,
310: simulator, 320: controller,
410: Input/Output Unit, 420: Planning Unit, 430: Simulation Unit,
431: Virtual controller, 440: Evaluation unit

Claims (13)

A servo system that performs visual servoing and has a sensor that captures and detects an object and a robot equipped with an end effector,
a processing device that determines a servo trajectory, which is a trajectory of the robot from an initial posture to a target posture, and a start posture, which is a posture of the robot at a point in the middle of the servo trajectory at which visual servoing of the robot is started;
A controller for the robot,
The processing device includes:
an acquisition means for acquiring information on the initial posture and information on the target posture of the robot;
1) determining a plurality of candidates for the start posture, and 2) generating a plurality of trajectories of the end effector when the posture of the robot changes from the initial posture to the target posture via any one of the plurality of candidates;
a determination means for determining, as the servo trajectory, a candidate having the highest evaluation value regarding the accuracy or speed of the visual servo from among a plurality of candidates of the servo trajectory corresponding to each of a plurality of trajectories of the end effector, and determining, as the starting posture, a candidate of a starting posture corresponding to the servo trajectory,
The controller:
A control is performed to change the posture of the robot according to the servo trajectory determined by the processing device,
A servo system characterized by controlling the posture of the robot without executing the visual servo during a period until the posture of the robot changes from the initial posture to the start posture, and starting the visual servo when the posture of the robot becomes the start posture. the planning means determines a plurality of candidates for the start posture from among postures between the initial posture and the target posture;
2. The servo system according to claim 1 . The servo trajectory includes at least one of a trajectory of the sensor, a trajectory of a relative position change between the object and the end effector, and a trajectory of the object in an image captured by the sensor.
3. The servo system according to claim 1 or 2. a simulation means for generating a plurality of candidates of the servo trajectory corresponding to each of a plurality of trajectories of the end effector, and simulating visual servoing according to each of the plurality of candidates of the servo trajectory;
The simulation means corrects each of a plurality of candidates for a trajectory of the servo system based on a simulation result;
The determining means determines the servo trajectory from among a plurality of candidates of the corrected servo trajectory.
4. The servo system according to claim 1, wherein the first and second electrodes are arranged in a first direction. The controller includes a first controller and a second controller,
The first controller determines a position and orientation of the end effector corresponding to each of a plurality of trajectories of the end effector;
the second controller controls a posture of the robot to realize the position and posture of the end effector determined by the first controller;
the simulation means has a third controller that emulates the first controller and a fourth controller that emulates the second controller, and simulates the visual servo using the third controller and the fourth controller;
5. The servo system according to claim 4. and further comprising an evaluation means for evaluating each of the plurality of candidates for the servo trajectory.
6. The servo system according to claim 1, wherein the first and second electrodes are arranged in a first direction. the evaluation means calculates an evaluation value for each of the plurality of candidates for the servo trajectory based on a task score related to the movement of the robot and a sensing score related to the detection of the object when the visual servoing is executed according to each of the plurality of candidates for the servo trajectory.
7. The servo system according to claim 6. the task score is a score based on at least one of a time required for the robot to move and a difference between a posture of the robot at a time point when the visual servoing ends and the target posture;
8. The servo system according to claim 7. the servo system is capable of detecting the object;
The sensing score is based on at least one of a total number of expected invalid detections during execution of visual servoing, a detectable range in which the servo system can detect an object and a distance between the object and the detectable range, and a total number of expected detection errors.
9. The servo system according to claim 7 or 8. The expected total number of invalid detections is the number of times that the sensor performs imaging, in which at least one of the following occurs: a first percentage or more of the object is not included in the field of view or the detectable range of the sensor; and another object overlaps an area of the object that is a second percentage or more of the entire area in the image captured by the sensor.
10. The servo system according to claim 9. the planning means determines a plurality of candidates for the start posture from a plurality of postures of the robot corresponding to a plurality of positions and postures of the end effector located within a predetermined range,
11. The servo system according to claim 1 . the planning means selects a plurality of candidates for the start posture from a plurality of postures of the robot corresponding to a plurality of positions and postures of the sensor in which no other object is disposed between the sensor and the target object when the sensor captures an image and the target object is included in a field of view of the sensor;
12. The servo system according to claim 11. A servo system that performs visual servoing and has a sensor that captures and detects an object and a robot equipped with an end effector,
A determination process for determining a servo trajectory, which is a trajectory of the robot from an initial posture to a target posture, and a start posture, which is a posture of the robot at a point in the middle of the servo trajectory and at a point when visual servoing of the robot is started;
and a control process for controlling the robot to change its posture in accordance with the determined servo trajectory,
The determination process includes:
an acquisition step of acquiring information on the initial posture and information on the target posture of the robot;
1) determining a plurality of candidates for the start posture; and 2) generating a plurality of trajectories of the end effector when the posture of the robot changes from the initial posture to the target posture via any one of the plurality of candidates.
a determining step of determining, as the servo trajectory, a candidate having the highest evaluation value regarding the accuracy or speed of the visual servo from among a plurality of candidates of the servo trajectory corresponding to each of a plurality of trajectories of the end effector, and determining, as the starting posture, a candidate of a starting posture corresponding to the servo trajectory,
The control process controls the robot's posture without executing visual servoing during the period until the robot's posture changes from the initial posture to the start posture, and starts the visual servoing when the robot's posture becomes the start posture.

Images