JP7608851B2 - Control system, motion planning device, control device, motion planning and control method, motion planning method, and control method - Google Patents

Description

The present disclosure relates to a control system, a motion planning device, a control device, a motion planning and control method, a motion planning method, and a control method.

A wide variety of parts are required for product assembly work, and in the product manufacturing process, assembly procedures must be created taking into account the properties of these diverse parts. Against this background, technology for creating part assembly procedures has been proposed.

For example, there is a technology for generating an assembly sequence taking into account the type of parts (see Patent Document 1). This technology analyzes the adjacent relationships between components using a predetermined analysis method according to the type of parts, and generates an assembly sequence for the components.

There is also a technology for automating the teaching of an appropriate hand selection task that takes into account compatibility with the part to be grasped (see Patent Document 2). This technology uses a part information DB that stores information about the part and a hand information DB that stores information about one or more hands to generate teaching information for selecting a hand to grasp the part, optimizing the hand selection to minimize assembly time.

There is also a technique for generating an appropriate assembly sequence (see Patent Document 3). In this technique, disassembly-possible components are identified from an assembly, the components are treated as nodes, edges are drawn between the nodes that are in a geometrically constrained relationship, and a disassembly graph is generated in which an evaluation value is assigned to each of the nodes and edges. The disassembly graph is then used to determine the disassembly sequence of the assembly, and an assembly sequence is generated.

There is also technology for generating movements for robots performing assembly work.

For example, research is being conducted into a method of using machine learning to teach a robot how to perform a task (see Non-Patent Document 1). This technology proposes a method of using reinforcement learning to teach a robot how to grasp an object based on image data obtained from a camera. This method makes it possible to automate at least part of the process of teaching a robot how to grasp an object.

There is also technology related to a versatile control device (see Patent Document 4). This technology involves determining a sequence that realizes the final target relative relationship amount from the relative relationship amount at the time when control of the operation is started, and generating a control command to change the relative relationship amount to the relative relationship amount in the next transition target state.

JP 2019-209446 A International Publication No. 2015/178377 International Publication No. 2015/177855 JP 2020-082332 A

Dmitry Kalashnikov, et al. "QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation" arXiv preprint arXiv:1806.10293, 2018.

However, the above-mentioned technology for creating assembly procedures is merely a method for creating procedure manuals that show the flow of work, and does not take into account how robots operate to assemble parts. Therefore, a method for creating assembly procedures that takes into account the operation of robots is required.

In addition, the above-mentioned technology for generating robot behavior has a drawback in that it may not be possible to properly reflect the behavior intended by the administrator who manages the robot when machine learning is introduced. For example, if the behavior generated by a model learned by machine learning differs from the behavior intended by the administrator, it is difficult to change the model because it takes time and cost to retrain the model to generate the intended behavior. Furthermore, even if the behavior is the same, the task intention may differ. Furthermore, there may be cases where the input and output do not correspond one-to-one. Therefore, it is not possible to uniquely generate robot behavior. Furthermore, although in machine learning, a model is often learned by learning one task at a time, in reality, the conditions for each task are often given by the administrator from a combination of multiple tasks. Therefore, it is difficult to learn a model in advance while taking into account these numerous conditions.

Therefore, it is desirable to create an assembly procedure that allows a manager to easily manage the robot's motion plan for each process. It is also desirable to control the robot's motion using the data from the created procedure.

The present disclosure has been made in consideration of the above circumstances, and aims to provide a control system for appropriately and efficiently managing the work process of a robot and for controlling the robot, as well as a motion planning method and a control method. It also aims to provide a motion planning device and a control device suitable for configuring such a control system.

A control system according to a first aspect of the present disclosure is a control system including a motion planning device and a control device, the motion planning device including a transition creation unit that creates first unit state transition data representing a transition from a state in which a first element, which is a part or assembly, and a second element, which is a part or assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled, a gripping data identification unit that identifies first gripping data that is a planned value of the relative position of the end effector and the first gripping object when the end effector of the robot grips the first element or the second element, which is a first gripping object, for assembly, and a final portion of the relative trajectory of the first element and the second element for performing the transition, the initial position of the first element and the second element. The control device includes an acquisition unit that acquires the first unit state transition data, the first gripping data, and the first guide trajectory, and acquires from a first sensor first observation data that observes the position of the first element and the position of the second element, and an execution unit that executes a first task to complete a state in which the third element is assembled in the first unit state transition data by gripping the first gripping object with the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory in its final portion.

The motion planning device according to the second aspect of the present disclosure includes a transition creation unit that creates first unit state transition data representing a transition from a state in which a first element, which is a part or assembly, and a second element, which is a part or assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled; a gripping data identification unit that identifies first gripping data that is a planned value of the relative position of the end effector and the first gripping object when the end effector of the robot grips the first element or the second element, which is a first gripping object, for assembly; and a guide trajectory creation unit that creates a first guide trajectory that is the final part of the relative trajectories of the first element and the second element for performing the transition and is a common target relative trajectory regardless of the initial relative positions of the first element and the second element.

The control device according to the third aspect of the present disclosure includes: first unit state transition data representing a transition from a state in which a first element, which is a part or an assembly, and a second element, which is a part or an assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled; first gripping data, which is a planned value of the relative position of the end effector and the first gripping object when the end effector of the robot grips the first element or the second element, which is a first gripping object, for assembly; and a final portion of the relative trajectory of the first element and the second element for performing the transition, which is an initial value of the first element and the second element. The system includes an acquisition unit that acquires a first guide trajectory, which is a common target relative trajectory regardless of the relative positions, and acquires first observation data that observes the positions of the first element and the second element from a first sensor, and an execution unit that executes a first task that executes a first task to complete a state in which the third element is assembled in the first unit state transition data by gripping the first gripping object with the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion.

The control system, operation planning, and control method of the present invention make it possible to appropriately and efficiently manage the robot's work process and control the robot.

FIG. 2 is a diagram showing the configuration of a control system for controlling a robot according to the present embodiment. FIG. 1 is a diagram showing a schematic configuration of a robot as an example of an object to be controlled. FIG. 2 is a block diagram showing the hardware configuration of the motion planning device and a control device. 2 is a block diagram showing an example of a functional configuration of a motion planning apparatus and a control apparatus. FIG. 11 is a diagram showing an example of unit state transition data created in the motion planning device; FIG. 1A to 1D are diagrams illustrating an example of an assembly procedure and a disassembly procedure using parts A to D. FIG. 11 is a diagram illustrating an example of a relative position of gripping data. 13 is an example of a relative trajectory recorded when an assembly of parts A to D is disassembled into an assembly of part D and parts A to C. FIG. 13 is a diagram showing an example of a guide path. FIG. 2 is a sequence diagram showing a processing flow of the control system according to the present embodiment. 13 is a flowchart showing a flow of a motion planning process of the motion planning device.

Below, an example of an embodiment of the present disclosure will be described with reference to the drawings. Note that the same reference symbols are used for identical or equivalent components and parts in each drawing. Also, the dimensional ratios in the drawings have been exaggerated for the convenience of explanation and may differ from the actual ratios.

Figure 1 is a diagram showing the configuration of a control system for controlling a robot according to this embodiment. As shown in Figure 1, the control system 1 has a robot 10, a state observation sensor 14, a motion planning device 20, a state transition control unit 25, and a control device 30. Note that the state transition control unit 25 may be part of the motion planning device 20, may be part of the control device 30, or may be configured as a device independent of both the motion planning device 20 and the control device 30 as in this embodiment.

(robot)
2 and 3 are diagrams showing a schematic configuration of a robot 10 as an example of a control target. The robot 10 in this embodiment is a six-axis vertical articulated robot, and an end effector 12 is provided at the tip 11a of an arm 11. The robot 10 performs an assembly task by gripping a part with the end effector 12. In the example of FIG. 3, the end effector 12 is configured to have a pair of hands of a clamping portion 12a, but the end effector 12 may be a suction pad to suck a part. In this specification, gripping a part includes sucking a part.

As shown in FIG. 2, the robot 10 has an arm 11 with six degrees of freedom equipped with joints J1 to J6. Each of the joints J1 to J6 connects the links together so that they can rotate in the directions of the arrows C1 to C6 by a motor (not shown). A gripper is connected to the tip of the arm 11 as an end effector 12. Here, a vertical multi-joint robot is used as an example, but a horizontal multi-joint robot (SCARA robot) may also be used. Also, a six-axis robot is used as an example, but a multi-joint robot with other degrees of freedom such as five axes or seven axes may also be used, or a parallel link robot may also be used.

(Status observation sensor)
The state observation sensor 14 observes the state of the robot 10 and outputs the observed data as state observation data. For example, an encoder for the joints of the robot 10, a visual sensor (camera), motion capture, a force-related sensor, etc. are used as the state observation sensor 14. The position and posture of the tip 11a of the arm 11 can be specified from the angle of each joint as the state of the robot 10, and the posture of the part (work object) can be estimated from the visual sensor and/or the force-related sensor. When a marker for motion capture is attached to the end effector 12, the position and posture of the end effector 12 can be specified as the state of the robot 10, and the posture of the part (work object) can be estimated from the position and posture of the end effector 12.

The term "force-related sensor" is a general term for force sensors and torque sensors, and also includes tactile sensors when the sensor is provided at a location that comes into contact with a part. The force-related sensor may be provided on the surface of the part where the end effector 12 grips a part or on a joint within the end effector 12 so as to detect the force that the end effector of the robot 10 receives from the part. The force-related sensor is, for example, a sensor that detects one-axis, three-axis, or six-axis forces of one or multiple elements as the state of the robot 10. By using a force-related sensor, it is possible to more accurately grasp how the end effector 12 is gripping the part, i.e., the posture of the part, and to enable appropriate control.

The position and posture of the end effector 12 itself and the part being held by the end effector 12 can also be detected as the state of the robot 10 using a visual sensor.

In this way, the state of the end effector 12 and the gripped part can be detected by the various sensors that are the state observation sensor 14. Furthermore, the detection results of the various sensors can be acquired as state observation data. The state observation sensor 14 is an example of a first sensor. Furthermore, the first sensor may be one sensor, or may be a general term for sensors in which the sensor that observes the first element and the sensor that observes the second element are different from each other. Furthermore, the state observation sensor 14 is an example of a second sensor. The second sensor may be the same sensor as the first sensor.

(Motion planning device/control device)
Next, the configurations of the motion planning device 20 and the control device 30 will be described.

Figure 3 is a block diagram showing the hardware configuration of the motion planning device 20 and the control device 30 according to this embodiment. The motion planning device 20 and the control device 30 can be realized with the same hardware configuration. The motion planning device 20 has a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, a storage 20D, an input unit 20E, a display unit 20F, and a communication interface (I/F) 20G. Each component is connected to each other so that they can communicate with each other via a bus 20H. The control device 30 has a CPU 30A, a ROM 30B, a RAM 30C, a storage 30D, an input unit 30E, a display unit 30F, and a communication I/F 30G. Each component is connected to each other so that they can communicate with each other via a bus 30H. The case of the motion planning device 20 will be described below.

In this embodiment, programs are stored in ROM 20B or storage 20D. CPU 20A is a central processing unit that executes various programs and controls each component. That is, CPU 20A reads out programs from ROM 20B or storage 20D, and executes the programs using RAM 20C as a working area. CPU 20A controls each of the above components and performs various arithmetic processing according to the programs recorded in ROM 20B or storage 20D.

ROM 20B stores various programs and various data. RAM 20C temporarily stores programs or data as a working area. Storage 20D is composed of a HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory, and stores various programs including the operating system, and various data.

The input unit 20E includes a keyboard and a pointing device such as a mouse, and is used to perform various inputs. The display unit 20F is, for example, a liquid crystal display, and displays various information. The display unit 20F may be a touch panel type and function as the input unit 20E.

The communication interface (I/F) 20G is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark).

Figure 4 is a block diagram showing an example of the functional configuration of the motion planning device 20 and the control device 30. Note that the motion planning device 20 and the control device 30 may be configured as an integrated unit.

As shown in FIG. 4, the motion planning device 20 has, as its functional components, a transition creation unit 110, a grip data identification unit 112, and a guide trajectory creation unit 114. Each functional component of the motion planning device 20 is realized when the CPU 20A reads out a program stored in the ROM 20B or storage 20D, expands it in the RAM 20C, and executes it. The control device 30 has, as its functional components, an acquisition unit 130 and an execution unit 132. Each functional component of the control device 30 is realized when the CPU 30A reads out a program stored in the ROM 30B or storage 30D, expands it in the RAM 30C, and executes it.

Now, the state transition data created by the motion planning device 20 of this embodiment will be described. FIG. 5 is a diagram showing an example of unit state transition data created by the motion planning device 20. In this embodiment, the assembly process by the work of the robot 10 is represented by a state transition diagram. As shown in FIG. 5, the entire state transition diagram 201 is decomposed into element units of parts or assemblies, and data representing the transitions of the elements is represented as unit state transition data (202A and 202B). The unit state transition data is a partial element of the composite state transition data that includes each of the unit state transition data, and by decomposing into unit state transition data, efficient management of control data such as gripping data and guide trajectories, which will be described later, becomes possible.

The following describes each processing unit of the motion planning device 20. The motion planning device 20 creates each unit state transition data as a motion plan, and accepts registration information required for creating the unit state transition data by input from the user. The registration information includes various information such as part information (part ID, type, etc.), CAD information for each type of end effector 12, assembly procedures (including disassembly procedures), and part gripping positions (including postures). The assembly procedures include the relative trajectories of the parts recorded during assembly (or disassembly) in the simulation.

The transition creation unit 110 creates composite state transition data that includes each of the unit state transition data linked together by their common elements. Each unit state transition data is created from an assembly procedure in the registration information.

In the example of FIG. 5, the transition creation unit 110 creates first unit state transition data 202A that represents a transition from a state in which a first element A and a second element B exist independently to a state in which a third element A', which is an assembly made up of the first element and the second element, is assembled. The first element A is a part or an assembly. The second element B is also a part or an assembly.

Next, the transition creation unit 110 creates the next second unit state transition data 202B using the third element A'. The second unit state transition data 202B represents unit state transition data that indicates the next assembly process after the first unit state transition data 202A transitions to a state in which the third element A' is being assembled. The transition creation unit 110 creates second unit state transition data 202B that represents a transition from a state in which the third element A' and the fourth element C exist independently to a state in which the fifth element A'', which is an assembly consisting of the third element A' and the fourth element C, is being assembled.

The composite state transition data 201 created by the transition creation unit 110 is in a format in which the first unit state transition data 202A and the second unit state transition data 202B are linked by a third element A', which is a common element of the first unit state transition data 202A and the second unit state transition data 202B.

In the above explanation, unit state transition data 202A is the first unit state transition data, but which unit state transition data is the first unit state transition data is arbitrary. For example, unit state transition data 202B may be the first unit state transition data, and the next unit state transition data may be the second unit state transition data. The second unit state transition data is data that represents the creation of a new assembly element by combining the element assembled by the transition represented by the first unit state transition data with another element.

The transition creation unit 110 may create only the unit state transition data 202A and 202B without creating the compound state transition data 201.

Before explaining the grip data specification unit 112 and the guide path creation unit 114, a specific example of an assembly procedure corresponding to the composite state transition data will be described. FIG. 6 is a diagram showing an example of an assembly procedure and a disassembly procedure using parts A to D. When applied to state transitions, each of (1) to (4) corresponds to unit state transition data, and data in a format in which all processes in (1) to (4) are linked corresponds to the composite state transition data. When the work process of assembling parts A and B in (1) is applied to the above-mentioned first unit state transition data, parts A and B correspond to the first element and the second element, respectively, and the assembly of parts A and B corresponds to the third element. (2) to (4) can also be applied to the first unit state transition data in the same manner. Furthermore, when the work process in (2) is applied to the second unit state transition data, the assembly of parts A and B corresponds to the third element, part C corresponds to the fourth element, and the assembly of parts A to C corresponds to the fifth element. (3) and (4) can also be applied to the second unit state transition data in a similar manner. Below, the grip data identification unit 112 and the guide trajectory creation unit 114 will be explained using the assembly procedure for parts A to D as an example.

The grip data identification unit 112 identifies grip data for each unit state transition data based on the grip position of the part in the registration information. The grip data is a planned value of the relative position of the end effector and the grip object when the end effector 12 of the robot 10 grips the grip object for assembly. The grip data identification unit 112 identifies first grip data when the end effector 12 grips the first element or second element, which is the first grip object. The grip data identification unit 112 also identifies second grip data when the end effector 12 of the robot 10 grips the third element or fourth element, which is the second grip object. The relative position in the grip data will be explained below. Here, the other element relative to the element set as the grip object becomes the target object.

Figure 7 shows an example of relative positions in gripping data. In Figure 7(A), the relative position R1 of the end effector 12 with respect to part A is represented as an arrow when the end effector 12 is a suction pad. Figure 7(A) shows the relative position R1 of the end effector 12 with respect to part A when the end effector 12 is a suction pad. Figure 7(B) shows the relative position R2 of the end effector 12 with respect to part B when the end effector 12 is a gripper (hand) with a pair of clamping parts. Parts C and D are represented in a similar manner.

The relative position is calculated by the following formula (1).
(T ^obj→tcp ) ^-1 =T ^tcp→obj ...(1)
The left side represents the part coordinate system, and the right side represents the relative position (position and orientation) from the end effector 12. The gripping data is calculated, for example, as a planned value of the target relative position of the object to be gripped and CAD data with an identification ID of the end effector 12 used during gripping. The planned value of the relative position may be input by the user, who is the administrator, by including it in the gripping position of the part in the registered information (the position on the surface of the part to be gripped), or may be automatically calculated by an existing method such as a gripping plan.

The guide trajectory creation unit 114 creates a guide trajectory for each piece of unit state transition data, based on the assembly procedure in the registration information. The guide trajectory is the final part of the relative trajectory between the element that is the object to be grasped for performing the state transition and the element that is the object to be incorporated, and is a trajectory that is a common target relative trajectory regardless of the initial relative positions between these elements. The guide trajectory creation unit 114 creates a first guide trajectory for the assembly of the first element and the second element. The guide trajectory creation unit 114 also creates a second guide trajectory for the assembly of the third element and the fourth element. Note that, in addition to the case where the relative trajectory is represented by position information, there are also cases where the trajectory is represented explicitly by velocity information without including position information. When the trajectory is represented by velocity information, it is represented by a vector quantity including directional information.

The first guide trajectory can be created as the initial part of the relative trajectory between the first and second elements recorded when the third element is disassembled into the first and second elements, or as the final part of the relative trajectory between the first and second elements recorded when the third element is assembled from the first and second elements. An example of a method for creating a guide trajectory is described below.

Figure 8 shows an example of a relative trajectory recorded when an assembly of parts A to D is disassembled into an assembly of parts D and parts A to C. In other words, this is an example of the initial portion of the relative trajectory of the first and second elements recorded when the third element is disassembled. In Figure 8, the relative trajectory G1 of part D is shown, and the reverse trajectory of this relative trajectory G1 can be used as the guide trajectory when grasping and assembling part D. Such a relative trajectory can be calculated when generating the procedure for disassembling from the target state. As a disassembly procedure, a bounding box (BB) is set for each process, and the disassembly of that process is completed when the bounding box no longer overlaps with the parts that have been moved during disassembly.

Figure 9 shows an example of a guide path. Figure 9 (A) shows a guide path (G2) when part A is grasped and lifted. Figure 9 (B) shows a guide path (G3) when part B is attached to part A.

The state transition control unit 25 acquires the composite state transition data or the unit state transition data, the grip data, and the guide trajectory from the motion planning device 20. When the state transition control unit 25 acquires the composite state transition data, it breaks it down into unit state transition data. When the state transition control unit 25 receives a notification from the control device 30 that one state transition has been completed, it instructs the control device 30 to execute the next state transition. In addition, the state transition control unit 25 provides the unit state transition data, the grip data, and the guide trajectory to the control device 30 in advance all at once or each time a state transition occurs so that the control device 30 can use the unit state transition data, the grip data, and the guide trajectory to execute the state transition. The state transition control unit 25 may be included in the motion planning device 20, may be included in the control device 30, or may be a device different from either of them. In addition, the entire control system including the motion planning device 20, the control device 30, and the state transition control unit 25 may be a single device.

The following describes each processing unit of the control device 30 using the state transitions in Figure 5 as an example. Here, one state transition is called a task.

The acquisition unit 130 acquires the unit state transition data 202A (first unit state transition data) and the corresponding grip data (first grip data) and guide path (first guide path) from the state transition control unit 25. The acquisition unit 130 also acquires observation data (first observation data) that observes the position of element A (first element) and the position of element B (second element) from the state observation sensor 14 (first sensor). Note that the positions of element A and element B include postures. In accordance with the progress of the task, the acquisition unit 130 acquires the unit state transition data 202B (second unit state transition data) and the corresponding grip data (second grip data) and guide path (second guide path) from the state transition control unit 25. The acquisition unit 130 acquires observation data (second observation data) that observes the position of element A' (third element) and the position of element C (fourth element) from the state observation sensor 14 (second sensor).

The execution unit 132 uses the unit state transition data 202A output from the state transition control unit 25 to execute a task (first task) which is the state transition represented by the unit state transition data 202A. Specifically, the execution unit 132 causes the end effector 12 to grasp a grasp object (first grasp object) which is one of elements A and B based on the observation data and grasping data for elements A and B, and moves the grasp object relative to the other element along a target relative trajectory which includes the guide trajectory in its final portion.

To explain in more detail, the execution unit 132 grasps the positions of elements A and B in the robot coordinate system from the observation data. Assuming that element A is to be grasped, the start position of the guide path in the robot coordinate system is grasped from the position of element B and the guide path. The operational movement amount of the robot 10 to the start position of the guide path is calculated, and element A is grasped by the end effector 12 based on the grasping data. Element A grasped by the end effector 12 is moved to the start position of the guide path, and then moved along the guide path. The execution unit 132 may execute the above operations by appropriately correcting the observation error, the grasping position error, and the directional error between the robot coordinate system and the guide path.

Here, an example of movement using a guide trajectory will be described. The execution unit 132 generates a transition trajectory, which is a trajectory that moves a grasped object, which is one of elements A and B, from its initial position to the starting position of the guide trajectory or any position on the guide trajectory relative to the other element. In the example of FIG. 9(B) , the trajectory ST from grasping part B to the starting position of the guide trajectory G3 is an example of a relative trajectory. Note that the transition trajectory may be either a relative trajectory or a trajectory in the robot coordinate system.

The non-grasped element is assumed to be fixed as the target object to be incorporated, but it may be grasped and moved by the end effector 12 of another robot 10. Also, if movement occurs due to contact between parts during the task, it may be observed again.

In addition, the execution unit 1232 executes a task (second task) of assembling element A'' (fifth element) from element A' (third element) and element C (fourth element) in Figure 5 using the next unit state transition data (second unit state transition data) output from the state transition control unit 25 according to the progress of the task, in the same way as the preceding task.

Next, the operation of the control system 1 will be explained.

Figure 10 is a sequence diagram showing the processing flow of the control system 1 of this embodiment. The CPU 20A performs the motion planning process by functioning as each part of the motion planning device 20, and the CPU 30A performs the control process by functioning as each part of the control device 30.

In step S100, the motion planning device 20 creates composite state transition data and control data by linking each of the unit state transition data. The control data is each of the grip data and each of the guide trajectories corresponding to each of the created unit state transition data. The processing of step S100 will be described in detail later.

In step S102, the motion planning device 20 outputs the compound state transition data and the control data to the state transition control unit 25. The state transition control unit 25 divides the compound state transition data into unit state transition data.

In step S104, the state transition control unit 25 receives a task start instruction and starts the task to be processed. When the task start instruction is received, the first task to be processed is started, and the task to be processed is updated according to the progress of the task. The task start instruction may be received by the control device 30.

In step S106, the state transition control unit 25 notifies the control device 30 of the start of the task.

In step S108, the state transition control unit 25 outputs unit state transition data corresponding to the task to be processed and control data corresponding to the unit state transition data to the control device 30.

In step S110, when the acquisition unit 130 of the control device 30 receives a notification of the start of the task, it acquires the observation data observed by the state observation sensor 14.

In step S112, the execution unit 132 of the control device 30 calculates the transition trajectory of the robot 10 to the starting position of the guide trajectory based on the observation data and the gripping data.

In step S114, the execution unit 132 of the control device 30 executes the task to be processed. That is, the end effector 12 grasps the element to be grasped based on the grasping data, and moves the grasped object relative to the other element along the calculated transition trajectory and guide trajectory.

In step S116, the execution unit 132 of the control device 30 determines whether the end point of the guide path has been reached in the execution of the task. If it is determined that the end point has been reached, the process proceeds to step S118, and if it is determined that the end point has not been reached, the determination in step S116 is repeated.

In step S118, the execution unit 132 of the control device 30 notifies the state transition control unit 25 of task completion.

In step S120, the state transition control unit 25 determines whether processing has been completed up to the final state of the compound state transition data. If it is determined that processing has been completed up to the final state, the process proceeds to step S124. If it is determined that processing has not been completed up to the final state, the process proceeds to step S122.

In step S122, the state transition control unit 25 updates the task being processed, and the process returns to step S108. Updating a task is a process of advancing the task being processed to the next task. In step S108 after the task update, the unit state transition data and control data (gripping data and guide trajectory) corresponding to the task being processed are output to the control device 30, and the subsequent processes are repeated.

In step S124, the state transition control unit 25 ends the task and terminates the processing of the control system 1.

Figure 11 is a flowchart showing the flow of the motion planning process of the motion planning device 20 in step S100.

In step S200, the CPU 20A accepts registration information input by the user. The registration information includes various information such as part information (part ID, type, etc.), CAD information for each type of end effector 12, assembly procedures, and part gripping positions (including postures).

In step S202, the transition creation unit 110 creates composite state transition data by linking each of the unit state transition data based on the assembly procedure in the registration information.

In step S204, the grip data identification unit 112 identifies grip data for each unit state transition data based on the grip position of the part in the registration information.

In step S206, the guide trajectory creation unit 114 creates a guide trajectory for each unit state transition data based on the assembly procedure (recorded relative trajectories of parts) in the registration information.

In step S208, the CPU 20A displays each of the created unit state transition data, the grip data, and the guide path on the display unit 20F. Note that the display here is not limited to the display unit 20F, and may be via another device.

In step S210, CPU 20A determines whether a correction instruction has been received from the user. If it is determined that a correction instruction has been received, the process proceeds to step S212, and if it is determined that a correction instruction has not been received, the process proceeds to step S102.

In step S212, the CPU 20A reflects the contents of the correction instruction in each of the unit state transition data, the grip data, and the guide trajectory. Correction instructions include, for example, correcting the state transition diagram of each of the unit state transition data, adding a waypoint, correcting the grip position of a part in the grip data, correcting the relative trajectory in the guide trajectory, etc. In the output of step S102, composite state transition data consisting of each of the final unit state transition data, and control data which is the grip data and the guide trajectory are output to the state transition control unit 25.

As described above, according to the control system 1 of this embodiment, composite state transition data and control data are created by linking each of the unit state transition data, and tasks are controlled by the unit state transition data. This makes it possible to appropriately and efficiently manage the work process of the robot.

In the above embodiment, the operation planning process or control process executed by the CPU by reading the software (program) may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and dedicated electrical circuits that are processors having a circuit configuration designed specifically to execute specific processes, such as ASICs (Application Specific Integrated Circuits). The operation planning process or control process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

In the above embodiment, the program is pre-stored (installed) in ROM 20B (30B) or storage 20D (30D), but this is not limiting. The program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

Reference Signs List 1 Robot system 10 Robot 11 Arm 12 Gripper 14 State observation sensor 20 Motion planning device 25 State transition control unit 30 Control device

Claims (12)

A control system including a motion planning device and a control device,
The motion planning device includes:
a transition creation unit that breaks down a part or an assembly into element units and creates first unit state transition data that represents a transition of elements from a state in which a first element, which is a part or an assembly, and a second element, which is a part or an assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled;
a gripping data specifying unit that specifies first gripping data that is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; and
a guide trajectory creation unit that creates a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and is a common target relative trajectory regardless of an initial relative position of the first element and the second element;
The control device includes:
an acquisition unit that acquires the first unit state transition data, the first gripping data, and the first guide trajectory, and acquires first observation data that observes a position of the first element and a position of the second element from a first sensor; and
an execution unit that executes a first task of gripping the first gripping object by the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion, to complete a state in which the third element is assembled in the first unit state transition data,
In the motion planning device,
The transition creation unit further creates second unit state transition data representing a transition of an element from a state in which the third element and a fourth element, which is a part or an assembly, exist independently to a state in which a fifth element, which is an assembly including the third element and the fourth element, is assembled;
the gripping data specifying unit further specifies second gripping data which is a planned value of a relative position between the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element which is a second gripping object for assembly;
The guide trajectory creation unit further creates a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition, and is a common target relative trajectory regardless of the initial relative positions of the third element and the fourth element;
The processor of the motion planning device displays each of the created first unit state transition data and the second unit state transition data, the first grip data and the second grip data, and the first guide trajectory and the second guide trajectory on a display unit,
When a processor of the motion planning device receives a correction instruction from a user, the processor executes at least one of the following corrections in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory;
In the control device,
the acquisition unit further acquires the second unit state transition data, the second gripping data, and the second guide trajectory in accordance with progress of the task, and acquires second observation data that observes a position of the third element and a position of the fourth element from a second sensor;
the execution unit, according to progress of the task, further executes a second task of gripping the second gripping object by the end effector based on the second observation data and the second gripping data, and moving the second gripping object, which is one of the third element and the fourth element, relative to the other element along a target relative trajectory that includes the second guide trajectory in a final portion, thereby completing a state in which the fifth element is assembled in the second unit state transition data.
Control system. The transition creation unit creates composite state transition data including the first unit state transition data and the second unit state transition data in a form in which common parts of each other are aggregated.
The control system of claim 1 . The control system further includes a state transition control unit,
the state transition control unit outputs the second unit state transition data to the execution unit of the control device after the first task is completed;
the execution unit of the control device executes the second task using the output second unit state transition data.
A control system according to claim 1 or 2 . A control system as described in any one of claims 1 to 3, wherein the execution unit of the control device generates a transition trajectory, which is a trajectory that moves the first grasped object, which is one of the first element and the second element, relative to the other element from its initial position to the starting position of the first guide trajectory or any position on the first guide trajectory. a transition creation unit that breaks down a part or an assembly into element units and creates first unit state transition data that represents a transition of elements from a state in which a first element, which is a part or an assembly, and a second element, which is a part or an assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled;
a gripping data specifying unit that specifies first gripping data that is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly;
a guide trajectory generation unit that generates a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and is a common target relative trajectory regardless of an initial relative position of the first element and the second element;
Including,
The transition creation unit further creates second unit state transition data representing a transition of an element from a state in which the third element and a fourth element, which is a part or an assembly, exist independently to a state in which a fifth element, which is an assembly including the third element and the fourth element, is assembled;
the gripping data specifying unit further specifies second gripping data which is a planned value of a relative position between the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element which is a second gripping object for assembly;
The guide trajectory creation unit further creates a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition, and is a common target relative trajectory regardless of the initial relative positions of the third element and the fourth element;
The processor displays the created first unit state transition data and the created second unit state transition data, the created first grip data and the created second grip data, and the created first guide trajectory and the created second guide trajectory on a display unit,
When the processor receives a correction instruction from a user, the processor executes at least one of the following corrections in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory.
Motion planning device. The transition creation unit creates composite state transition data including the first unit state transition data and the second unit state transition data in a form in which common parts of each other are aggregated.
The motion planning device according to claim 5 . 7. The motion planning device according to claim 5, wherein the first guide trajectory is an initial portion of a relative trajectory of the first element and the second element recorded when the third element is disassembled into the first element and the second element, or a final portion of the relative trajectory of the first element and the second element recorded when the third element is assembled from the first element and the second element. First unit state transition data representing a transition of elements from a state in which a first element, which is a part or assembly, and a second element, which is a part or assembly, exist independently, to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled, by breaking down a part or assembly into element units;
First gripping data is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; and
obtain a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and is set as a common target relative trajectory regardless of initial relative positions of the first element and the second element;
an acquisition unit that acquires first observation data that observes a position of the first element and a position of the second element from a first sensor;
an execution unit that executes a first task of gripping the first gripping object by the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion, to complete a state in which the third element is assembled in the first unit state transition data;
Including,
The acquisition unit further acquires second unit state transition data representing a transition from a state in which the third element and a fourth element which is a part or an assembly exist independently to a state in which a fifth element which is an assembly including the third element and the fourth element is assembled;
Second gripping data is a planned value of a relative position of the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element, which is a second gripping object, for assembly; and
obtaining a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition, and which is set as a common target relative trajectory regardless of initial relative positions of the third element and the fourth element;
acquiring second observation data from a second sensor that observes a position of the third element and a position of the fourth element;
When a correction instruction is received from a user at a device from which the data is acquired by the acquisition unit, at least one of the following corrections is executed in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory;
the execution unit, according to progress of the task, further executes a second task of gripping the second gripping object by the end effector based on the second observation data and the second gripping data, and moving the second gripping object, which is one of the third element and the fourth element, relative to the other element along a target relative trajectory that includes the second guide trajectory in a final portion, thereby completing a state in which the fifth element is assembled in the second unit state transition data.
Control device. The control device according to claim 8, wherein the execution unit generates a transition trajectory, which is a transition portion before the final portion of the relative trajectory, and which is a trajectory that moves the first grasped object, which is one of the first element and the second element, from its initial position to a starting position of the first guide trajectory or any position on the first guide trajectory relative to the other element, until the first grasped object transitions from its real space relative position to a target relative trajectory of the first guide trajectory. A motion planning and control method for performing motion planning and control, comprising:
In the motion plan,
disassembling the part or assembly into element units, and creating first unit state transition data representing a transition of the elements from a state in which a first element which is a part or assembly and a second element which is a part or assembly exist independently to a state in which a third element which is an assembly consisting of the first element and the second element is assembled;
Identifying first gripping data which is a planned value of a relative position between an end effector of a robot and a first gripping object when the end effector grasps the first element or the second element which is a first gripping object for assembly;
creating a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and which is a common target relative trajectory regardless of an initial relative position of the first element and the second element;
In the control,
acquiring the first unit state transition data, the first gripping data, and the first guide trajectory, and acquiring first observation data that observes a position of the first element and a position of the second element from a first sensor;
gripping the first gripping object by the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion, thereby executing a first task for completing a state in which the third element is assembled in the first unit state transition data ;
Further, in the motion plan,
In the creation of the transition data, second unit state transition data is created which represents a transition of an element from a state in which the third element and a fourth element which is a part or an assembly exist independently to a state in which a fifth element which is an assembly including the third element and the fourth element is assembled;
The identification of the gripping data further includes identifying second gripping data that is a planned value of a relative position of the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element, which is a second gripping object, for assembly;
The guide trajectory generation further includes generating a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition and is set as a common target relative trajectory regardless of the initial relative positions of the third element and the fourth element;
The processor displays the created first unit state transition data and the created second unit state transition data, the created first grip data and the created second grip data, and the created first guide trajectory and the created second guide trajectory on a display unit,
When the processor receives a correction instruction from a user, the processor executes at least one of the following corrections in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory;
Furthermore, in the control,
Regarding the acquisition, the second unit state transition data, the second gripping data, and the second guide trajectory are acquired according to a progress of a task, and second observation data observing a position of the third element and a position of the fourth element are acquired from a second sensor;
Regarding the execution, according to the progress of the task, a second task is executed to complete a state in which the fifth element is assembled in the second unit state transition data by gripping the second gripping object by the end effector based on the second observation data and the second gripping data, and moving the second gripping object, which is one of the third element and the fourth element, relative to the other element along a target relative trajectory that includes the second guide trajectory in a final portion thereof.
A method of planning and controlling operations for causing a computer to execute a process. disassembling the part or assembly into element units, and creating first unit state transition data representing a transition of the elements from a state in which a first element which is a part or assembly and a second element which is a part or assembly exist independently to a state in which a third element which is an assembly consisting of the first element and the second element is assembled;
Identifying first gripping data which is a planned value of a relative position between an end effector of a robot and a first gripping object when the end effector grasps the first element or the second element which is a first gripping object for assembly;
creating a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition , and which is a common target relative trajectory regardless of an initial relative position of the first element and the second element;
Furthermore, in the motion planning,
In the creation of the transition data, second unit state transition data is created which represents a transition of an element from a state in which the third element and a fourth element which is a part or an assembly exist independently to a state in which a fifth element which is an assembly including the third element and the fourth element is assembled;
The identification of the gripping data further includes identifying second gripping data that is a planned value of a relative position of the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element, which is a second gripping object, for assembly;
The guide trajectory generation further includes generating a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition and is set as a common target relative trajectory regardless of the initial relative positions of the third element and the fourth element;
The processor displays the created first unit state transition data and the created second unit state transition data, the created first grip data and the created second grip data, and the created first guide trajectory and the created second guide trajectory on a display unit,
When the processor receives a correction instruction from a user, the processor executes at least one of the following corrections in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory .
A motion planning method for causing a computer to execute a process. First unit state transition data representing a transition of elements from a state in which a first element, which is a part or assembly, and a second element, which is a part or assembly, exist independently, to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled, by breaking down a part or assembly into element units;
First gripping data is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; and
obtain a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and is set as a common target relative trajectory regardless of initial relative positions of the first element and the second element;
acquiring first observation data from a first sensor that observes a position of the first element and a position of the second element;
a first task is executed to complete a state in which the third element is assembled in the first unit state transition data by gripping the first gripping object with the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion thereof;
Furthermore, in the control,
second unit state transition data representing a transition from a state in which the third element and a fourth element, which is a part or an assembly, exist independently to a state in which a fifth element, which is an assembly including the third element and the fourth element, is assembled, with respect to the acquisition;
Second gripping data is a planned value of a relative position of the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element, which is a second gripping object, for assembly; and
obtaining a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition, and which is set as a common target relative trajectory regardless of initial relative positions of the third element and the fourth element;
acquiring second observation data from a second sensor that observes a position of the third element and a position of the fourth element;
When a correction instruction is received from a user by a device from which the data is acquired, at least one of the following corrections is executed in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory;
Regarding the execution, according to the progress of the task, a second task is executed to complete a state in which the fifth element is assembled in the second unit state transition data by gripping the second gripping object by the end effector based on the second observation data and the second gripping data, and moving the second gripping object, which is one of the third element and the fourth element, relative to the other element along a target relative trajectory that includes the second guide trajectory in a final portion thereof.
A control method for causing a computer to execute processing.