JP7834288B2 - Assembly system, control method, and program - Google Patents

Description

Applicable under Article 30, Paragraph 2 of the Patent Law (1) Publication Date: December 15, 2021 Publication: Proceedings of the 22nd SICE System Integration Division Conference Published by the SICE System Integration Division *Available via a participant-only address (viewable only by participants) (2) Date of Event: December 15, 2021 Meeting Name and Venue: The 22nd SICE System Integration Division Conference (Dates: December 15-17, 2021) Hosted by the SICE System Integration Division Online

This disclosure relates to an assembly system, a control method, and a program.

Patent Document 1 discloses a robot system in which a sensor detects changes in the contact state of the robot's end effector with respect to the workpiece, and estimates and corrects the amount of displacement of the end effector relative to the workpiece based on the changes in the contact state detected by the sensor.

Patent No. 6870122

However, the technology described in Patent Document 1 only compensates for the amount of misalignment based on the change in the contact state of the socket with the bolt head caused by the tactile movement of the end effector. Therefore, it is difficult to reproduce the fastening motion performed by a human. For example, it may take time for the bolt head to fit into the socket. Consequently, there is a need for assembly technology that enables smoother fastening.

This disclosure was made to solve the above-mentioned problems and aims to provide an assembly system, control method, and program that can be fastened more smoothly.

To solve the above problems, the assembly system according to this disclosure comprises a manipulator having an arm having a plurality of joints, an end effector that is moved by the arm and holds a nut and rotates the nut around an axis, a first sensor that acquires the joint angle of each of the joints, and a second sensor that acquires the contact force and torque transmitted from the end effector when the nut comes into contact with a bolt provided on the object to be constructed, and a control device that controls the manipulator, wherein the control device acquires the joint angle of each of the joints, the contact force and torque from the first The system comprises: an acquisition unit that acquires data at predetermined intervals; a calculation unit that calculates the position coordinates and orientation angle of the nut based on the acquired joint angles of each of the joints; and a control unit that controls the end effector using a first trained model that, upon receiving the position coordinates and orientation angle of the nut at a first time point, and the contact force and torque at the first time point, outputs command values indicating the position coordinates and orientation angle of the nut at a second time point, which is later than the first time point and approaches the target position coordinates and orientation angle of the nut indicating that the nut is fastened to the bolt.

The control method according to this disclosure is a control method for controlling a manipulator having an arm with a plurality of joints, an end effector moved by the arm and holding a nut and rotating the nut around an axis, a first sensor for acquiring the joint angle of each of the joints, and a second sensor for acquiring the contact force and torque transmitted from the end effector, the method comprising: a step of acquiring the joint angle of each of the joints, the contact force and torque at a first predetermined period; a step of calculating the position coordinates and orientation angle of the nut based on the acquired joint angles of each of the joints; and a step of controlling the end effector using a first trained model that, upon inputting the position coordinates and orientation angle of the nut at a first time and the contact force and torque at the first time, outputs command values indicating the position coordinates and orientation angle of the nut at a second time after the first time, when the nut has approached the target position coordinates and target orientation angle of the nut indicating a state in which the nut is fastened to the bolt.

The program according to this disclosure causes the computer of a control device that controls a manipulator having an arm with multiple joints, an end effector moved by the arm and holding a nut and rotating the nut around an axis, a first sensor for acquiring the joint angle of each of the joints, and a second sensor for acquiring the contact force and torque transmitted from the end effector, to execute the following steps: acquiring the joint angle of each of the joints, the contact force and torque at a first predetermined period; calculating the position coordinates and orientation angle of the nut based on the acquired joint angles of each of the joints; and controlling the end effector using a first trained model that, upon inputting the position coordinates and orientation angle of the nut at a first time and the contact force and torque at the first time, outputs command values indicating the position coordinates and orientation angle of the nut at a second time point after the first time point, when the nut has approached the target position coordinates and orientation angle of the nut indicating that the nut is fastened to the bolt.

This disclosure provides an assembly system, control method, and program that enable smoother fastening.

This is a schematic diagram showing the overall configuration of the assembly system according to the first embodiment of this disclosure. This diagram shows an end effector, nut, and bolt arranged in a row according to the first embodiment of the present disclosure. This is a view of the inside of a nut runner according to the first embodiment of this disclosure, seen from the axial direction. This figure illustrates the movement of the nut holding member when the nut is held in the end effector according to the first embodiment of this disclosure. This figure illustrates the movement of the nut retaining member when removing a collar from an end effector according to the first embodiment of this disclosure. This figure schematically shows an example of an image captured by the imaging device according to the first embodiment of this disclosure. This is a functional block diagram of a control device according to the first embodiment of this disclosure. This figure conceptually illustrates the coordinate system in which the position coordinates of the nut according to the first embodiment of this disclosure are represented. This figure conceptually illustrates the contact force received by an end effector when a nut held by the end effector according to the first embodiment of this disclosure comes into contact with the bolt. This figure conceptually illustrates the correction values used by the correction unit according to the first embodiment of this disclosure for correcting contact force. This flowchart shows an example of the training steps for a first trained model according to the first embodiment of this disclosure. This figure shows an example of a direct teaching operation when teaching a second trained model according to the first embodiment of this disclosure. This flowchart shows an example of the operation of the control device according to the first embodiment of this disclosure. This is a schematic diagram showing the overall configuration of the assembly system according to the second embodiment of this disclosure. This is a schematic diagram showing the overall configuration of the assembly system according to the third embodiment of this disclosure. This is a schematic diagram showing the overall configuration of an assembly system according to a modified example of the third embodiment of the present disclosure. This is a hardware configuration diagram showing the configuration of a computer according to the embodiments of this disclosure.

The following describes the configuration for implementing the assembly system, referring to the attached drawings.

<First embodiment of the assembly system>
An assembly system is part of an automated system that operates in an assembly plant, for example, and assembles multiple parts into a single unit by controlling a manipulator. Hereinafter, the parts that the assembly system assembles will be referred to as the "object to be assembled". As shown in Figure 1, the object to be assembled 200 is divided into, for example, a first part 210 and a second part 220, and is placed on, for example, a workbench 300 (work desk). The assembly system 100 integrates the first part 210 and the second part 220 by controlling the manipulator 10. Examples of the object to be assembled 200 include a panel, which is one of the aircraft parts, and a panel, which is one of the ship parts. The assembly system 100 includes, for example, a manipulator 10 and a control device 30.

(Manipulator configuration)
In this embodiment, the manipulator 10 is a robot that moves the nut 202 by changing its overall posture while holding the nut 202 as a connector, and automatically fastens the nut 202 to a bolt 201, which is a connector provided on the object to be constructed 200. The manipulator 10 is a multi-joint robot equipped with multiple joints 110. In this embodiment, a six-axis multi-joint robot equipped with six joints 110 is described as an example of the manipulator 10. The manipulator 10 integrates the first part 210 and the second part 220 by fastening the nut 202 to a bolt 201 that is temporarily fixed to the object to be constructed 200 with adhesive or the like.

As shown in Figure 2, in this embodiment, the nut 202 handled by the manipulator 10 is a collar nut having a nut body 202a and a collar 202b integrally provided on the nut body 202a. The nut 202 has a nut hole 202h as a female thread portion formed across both the nut body 202a and the collar 202b. In this embodiment, the outer shell of the collar 202b is formed in a hexagonal prism shape. During the process of fastening the nut 202 to the bolt 201 by the manipulator 10, only the collar 202b is removed (collar cut), and the nut body 202a is fastened to the bolt 201. The bolt 201 is inserted through holes (not shown) provided in the first part 210 and the second part 220, which are the objects to be constructed 200, and protrudes from the second part 220, so that the male thread portion 201a is exposed. In this embodiment, the bolt 201 is, for example, a stud bolt (recessed bolt) without a head. The manipulator 10 fastens the bolt 202 by screwing the nut hole 202h of the nut 202 onto the exposed male threaded portion 201a of the bolt 201. Returning to Figure 1, the manipulator 10 includes, for example, an arm 11, an encoder 130, a force sensor 140, an end effector 12, and a base portion 15.

(arm)
The arm 11 forms the main body of the manipulator 10. The arm 11 is placed on a base 15 positioned on the ground. Specifically, one end of the arm 11 is fixed to the base 15. The arm 11 can move three-dimensionally around the base 15 by changing its orientation with the base 15 as the pivot point. The arm 11 has, for example, a plurality (six) of joints 110, a first connection part 118a, a second connection part 118b, and a third connection part 118c.

Each joint 110 is, for example, cylindrical in shape and contains a motor 117 inside to make the joint 110 rotatable. A servo motor is one example of a motor 117 in a joint 110. The motor 117 rotates the entire joint 110 around a rotation axis, with the motor 117 as the center of rotation. Hereinafter, the six joints 110 of the arm 11 will be referred to, in order from the side closest to the base 15, as "first joint 111," "second joint 112," "third joint 113," "fourth joint 114," "fifth joint 115," and "sixth joint 116." Furthermore, the motor 117 of the first joint 111 is referred to as "first motor 117a," the motor 117 of the second joint 112 as "second motor 117b," the motor 117 of the third joint 113 as "third motor 117c," the motor 117 of the fourth joint 114 as "fourth motor 117d," the motor 117 of the fifth joint 115 as "fifth motor 117e," and the motor 117 of the sixth joint 116 as "sixth motor 117f." The rotation of each motor 117 (first motor 117a to sixth motor 117f) is controlled by the control device 30 (described later).

The first joint portion 111 is fixed to the upper surface of the base portion 15. The first joint portion 111 corresponds to the above-mentioned end of the arm 11. The first motor 117a is rotatable about a first rotation axis O1, for example, as a pan axis extending in the vertical direction, and rotates the entire first joint portion 111 around the first rotation axis O1. The second joint portion 112 is provided so as to extend from the outer surface of the first joint portion 111 in an integral state with the first joint portion 111. The second motor 117b is rotatable about a second rotation axis O2, for example, as a tilt axis extending in the horizontal direction, and rotates the entire second joint portion 112 around the second rotation axis O2. The second joint portion 112 is provided with a first connecting portion 118a that extends in a direction intersecting the second rotation axis O2. The first connecting portion 118a is cylindrical and extends from the outer surface of the second joint portion 112. The third joint portion 113 is integrally provided at the end of the extension from the second joint portion 112 of the first connecting portion 118a. The third motor 117c is rotatable around a third rotation axis O3, for example, as a tilt axis extending in the horizontal direction, and rotates the entire third joint portion 113 around the third rotation axis O3. The third joint portion 113 is provided with a second connecting portion 118b that extends in a direction intersecting the third rotation axis O3. The second connecting portion 118b is cylindrical and extends from the outer surface of the third joint portion 113. The fourth joint portion 114 is integrally provided at the end of the extension from the third joint portion 113 of the second connecting portion 118b. The fourth motor 117d is rotatable around a fourth rotation axis O4, for example, as a tilt axis extending horizontally, and rotates the entire fourth joint 114 around the fourth rotation axis O4. The fifth joint 115 is provided integrally with the fourth joint 114. The fifth motor 117e is rotatable around a fifth rotation axis O5, for example, as a pan axis extending in a direction intersecting the fourth rotation axis O4, and rotates the entire fifth joint 115 around the fifth rotation axis O5. The fifth joint 115 is provided with a third connecting portion 118c that extends in a direction intersecting the fifth rotation axis O5. The third connecting portion 118c is cylindrical and extends from the outer surface of the fifth joint 115. The sixth joint 116 is provided integrally with the end of the third connecting portion 118c that extends from the fifth joint 115. The sixth joint 116 corresponds to the other end of the arm 11 opposite to the aforementioned end. The sixth motor 117f is rotatable around the sixth rotation axis O6, which acts as a pan axis extending parallel to, for example, the fifth rotation axis O5. The rotation of each of the aforementioned joints 110 by the motor 117 causes a three-dimensional change in the posture of the arm 11. Hereinafter, the sixth rotation axis O6 will be simply referred to as "axis Ar1". The direction in which axis Ar1 extends will be referred to as "axial direction Da", one side of axial direction Da will be simply referred to as "one side Da1", and the side opposite to one side Da1 will be referred to as "the other side Da2". The motor 117 of each joint 110 is an example of a controlled device S.

(Encoder)
The encoder 130 is a sensor capable of acquiring the rotation angle of the joint 110. Hereinafter, the rotation angles (θ1 to θ6) of each joint 110 (first joint 111 to sixth joint 116) will be referred to as "joint angles". For example, one encoder 130 is provided on each motor 117 of each joint 110. Hereinafter, the encoder 130 provided on the first motor 117a will be referred to as the "first encoder 131," the encoder 130 provided on the second motor 117b will be referred to as the "second encoder 132," the encoder 130 provided on the third motor 117c will be referred to as the "third encoder 133," the encoder 130 provided on the fourth motor 117d will be referred to as the "fourth encoder 134," the encoder 130 provided on the fifth motor 117e will be referred to as the "fifth encoder 135," and the encoder 130 provided on the sixth motor 117f will be referred to as the "sixth encoder 136." Each encoder 130 transmits the joint angle (sensor data) acquired from the motor 117 to the control device 30. The encoder 130 is an example of the first sensor 13. The first sensor 13 is connected to the control device 30 by wired or wireless communication.

(Force sensor)
The force sensor 140 is a load cell connected to the arm 11. Specifically, the force sensor 140 is fixed to the sixth joint 116 from one side Da1 in the axial direction Da. The force sensor 140 is provided with an end effector 12 (described later) on the opposite side from the sixth joint 116 (one side Da1 in the axial direction Da). The force sensor 140 can acquire data on the force transmitted from the end effector 12. Specifically, when the nut 202 held by the end effector 12 comes into contact with the bolt 201, the force sensor 140 can acquire the magnitude of the contact force transmitted to the force sensor 140 from one side Da1, which is the end effector 12 side, for each of the three axial components (Fx, Fy, Fz), and can also acquire the magnitude of the torque (Mx, My, Mz) acting around the axis of each contact force acting on each axis. Hereinafter, the contact force (unit: N) and torque (unit: N・m) acquired by the force sensor data may be collectively referred to as "force data." In other words, the force sensor 140 can acquire force data for six axes. The force sensor 140 transmits the acquired force data (sensor data) to the control device 30. The force sensor 140 is an example of the second sensor 14. The second sensor 14 is connected to the control device 30 by wired or wireless communication.

(End effector)
The end effector 12 holds the nut 202 and rotates the nut 202 around the rotation axis Ar. The rotation axis Ar is, for example, a virtual axis extending in the axial direction Da at a position offset from the axis Ar1. Note that the rotation axis Ar may coincide with the axis Ar1. In this embodiment, the end effector 12 is provided on the force sensor 140 from one side Da1. Therefore, the end effector 12 is moved by the arm 11 and fastens the held nut 202 to a bolt 201 provided on the object to be constructed 200. As shown in Figures 1 and 2, the end effector 12 includes, for example, an imaging device 20, a main body 120, a nut runner 121, a nut holding member 124, and an elastic member 128. In Figure 2, for illustrative purposes, the socket portion 123 of the nut runner 121 is shown in cross-section so that the inside is visible.

(Imaging device)
As shown in Figures 1, 2, and 3, the imaging device 20 is provided on the end effector 12 and generates an image of the object to be constructed 200. In this embodiment, the imaging device 20 is positioned inside the socket portion 123 of the nut runner 121 of the end effector 12, which will be described later, in a state where it cannot rotate relative to the socket portion 123. Specifically, as shown in Figure 2, the imaging device 20 is positioned inside the socket portion 123 (the housing space R described later) on the side closer to the base portion 122 than the housing portion 123r described later. The imaging device 20 has, for example, an imaging unit 21 and an illumination unit 22. The imaging unit 21 generates an image of the object to be constructed 200 through the nut hole 202h of the nut 202 held by the nut holding member 124 described later. The imaging unit 21 can be exemplified by a sensor module incorporating a semiconductor element (solid-state image sensor) such as a CCD image sensor or a CMOS image sensor. The imaging unit 21 may be replaced with a fiberscope or the like. The illumination unit 22 irradiates light onto the object to be constructed 200 through the nut hole 202h. The illumination unit 22 may be an LED (Light Emitting Diode), for example. Figure 6 is a schematic diagram showing an example of an image captured by the imaging device 20. In Figure 6, an example of an image of a bolt 201 provided on the object to be constructed 200, taken from a slightly oblique angle above, is schematically shown. In Figure 6, a bolt 201 with a hole formed around its centerline Ar2 (the reference numeral is not shown) is shown as an example, but the bolt 201 does not necessarily have to have a hole formed therein. As shown in Figure 6, the imaging device 20 can image the surface of the object to be constructed 200, the male threaded portion 201a of the bolt 201, and the end face 201e of the bolt 201 while irradiating light onto the object to be constructed 200 and the bolt 201 through the nut hole 202h. The imaging device 20 is connected to the control device 30 by wired or wireless communication. The imaging device 20 transmits the captured images to the control device 30.

(Main body)
The main body 120 is fixed to the force sensor 140. The socket portion 123 (described later, see Figure 2) of the nut runner 121 is inserted through the main body 120, and the main body 120 has a fastening motor 120a (see Figure 1) inside for rotating the socket portion 123 around the rotation axis Ar. The rotation of the fastening motor 120a of the main body 120 is controlled by the control device 30. The fastening motor 120a of the main body 120 is an example of a controlled device S.

(Nut Runner)
As shown in Figure 2, the nut runner 121 has, for example, a base portion 122 and a socket portion 123. The base portion 122 is, for example, a disc shape centered on the axis of rotation Ar and is fixed to the main body portion 120 from one side Da1. The base portion 122 has a front surface 122a facing the one side Da1. The socket portion 123 is provided so as to extend one side Da1 beyond the front surface 122a of the base portion 122 and has a cylindrical shape centered on the axis of rotation Ar. The socket portion 123 is inserted into the base portion 122 and the main body portion 120 and is rotatably held by a fastening motor 120a provided in the main body portion 120 (see Figure 1). The socket portion 123 has a cylindrical inner circumferential surface 123i that defines the interior and a cylindrical outer circumferential surface 123o that faces outward. Hereinafter, the space inside the socket portion 123 will be referred to as the "accommodation space R". The socket portion 123 has an opening 123e at its tip, which is located on one side Da1. In this embodiment, the opening 123e is formed in the shape of a perfect circle perpendicular to the axis of rotation Ar. The socket portion 123 also has a housing portion 123r that includes the opening 123e and is capable of accommodating at least a part of the nut 202. The housing portion 123r is located on one side Da1 in the housing space R.

As shown in Figures 2 and 3, the housing section 123r is composed of, for example, a plurality of tapered surfaces 123a, a plurality of fastening surfaces 123b, and a positioning surface 123c. Each tapered surface 123a is inclined with respect to the direction in which the opening 123e widens. Specifically, each tapered surface 123a is inclined with respect to the direction in which the opening 123e widens so that it approaches the rotation axis Ar as it moves toward the other side Da2. In this embodiment, six tapered surfaces 123a are arranged in a line around the rotation axis Ar, and adjacent tapered surfaces 123a around the rotation axis Ar are connected to each other. The region defined by the plurality of tapered surfaces 123a in the housing section 123r has a cross-section perpendicular to the rotation axis Ar that forms a regular hexagon, and has a tapered shape that becomes smaller as it moves toward the other side Da2.

Each fastening surface 123b is connected to the tapered surface 123a from the other side Da2, corresponding to one tapered surface 123a, and is positioned perpendicular to the direction in which the opening 123e widens. In this embodiment, the six fastening surfaces 123b are arranged around the rotation axis Ar, and adjacent fastening surfaces 123b around the rotation axis Ar are connected to each other. The region defined by the multiple fastening surfaces 123b in the housing portion 123r forms a hexagonal prism shape with a cross-section perpendicular to the rotation axis Ar that is constant in the axial direction Da. The region defined by the multiple fastening surfaces 123b is sized to accommodate, for example, the collar 202b of the nut 202.

The positioning surface 123c is the surface that connects the multiple (six) fastening surfaces 123b and the inner circumferential surface 123i of the socket portion 123, and faces one side Da1. In this embodiment, the positioning surface 123c is arranged parallel to the direction in which the opening 123e widens. That is, the positioning surface 123c is arranged perpendicular to each of the fastening surfaces 123b. The positioning surface 123c is the surface for positioning the nut 202 when the nut 202 is housed in the housing portion 123r. As shown in Figure 3, when viewed from one side Da1, the positioning surface 123c has a shape obtained by removing the circular part from the central part of a regular hexagon. The collar 202b of the nut 202 comes into contact with this positioning surface 123c from one side Da1, thereby positioning the nut 202. In other words, the positioning surface 123c positions the nut 202, preventing it from entering the housing space R on the other side Da2 beyond the housing portion 123r.

(Nut retaining member)
The nut holding member 124 holds the nut 202 so that when the collar 202b of the nut 202 is housed in the housing portion 123r of the nut runner 121, the nut 202 does not fall out of the housing portion 123r to one side Da1. Returning to Figure 2, the nut holding member 124 is provided on the outer circumference of the socket portion 123. In this embodiment, two nut holding members 124 are provided on the outer circumference of the socket portion 123 so as to sandwich the socket portion 123 between them. Alternatively, three or more nut holding members 124 may be arranged at equal intervals around the rotation axis Ar with respect to the socket portion 123. The nut holding member 124 has, for example, a pivot shaft 125, a first portion 126, and a second portion 127.

The pivot shaft 125 is rotatably mounted on the outer circumferential surface 123o of the socket portion 123. The first portion 126 is fixed to the pivot shaft 125 and is a rod-shaped member extending in a direction intersecting the rotation axis Ar. The second portion 127 has an extended portion 127a extending from the first portion 126 toward the tip side of the socket portion 123, and a claw portion 127b provided at the tip of the extended portion 127a and protruding from the extended portion 127a so as to approach the rotation axis Ar. The extended portion 127a is a rod-shaped member extending along the rotation axis Ar. The extended portion 127a, together with the first portion 126, forms an L-shape. The claw portion 127b approaches the rotation axis Ar from the extended portion 127a toward the other side Da2 and has a tapered shape as it approaches the rotation axis Ar. The claw portion 127b is positioned on one side Da1 of the opening 123e of the socket portion 123. Furthermore, as shown in Figures 2 and 3, the tip of the claw portion 127b is located closer to the axis of rotation Ar than to the fastening surface 123b. When the collar 202b of the nut 202 is housed in the housing portion 123r of the socket portion 123, the tip of the claw portion 127b contacts the side surface of the nut 202 due to the elastic force of the elastic member 128 (described later). Specifically, when the nut 202 is housed in the housing portion 123r, the nut 202 is gripped (restrained) by the two claw portions 127b.

(Elastic material)
The elastic member 128 is a spring provided between the base 122 of the nut runner 121 and the first portion 126 of the nut holding member 124. Specifically, one end of the elastic member 128 is fixed to the front surface 122a of the base 122, and the other end is fixed to the first portion 126. Therefore, the elastic member 128 connects the base 122 and the first portion 126 to each other. The elastic member 128 applies an elastic force to the first portion 126 toward the tip side (one side Da1) of the socket portion 123. In this embodiment, one elastic member 128 is provided between the base 122 and the first portion 126 so that it corresponds to one nut holding member 124. Alternatively, multiple elastic members 128 may be provided between the base 122 and the first portion 126 so that they correspond to one nut holding member 124. Furthermore, the elastic member 128 is not limited to a spring, as long as it can impart an elastic force to the first portion 126 toward the tip side (one side Da1) of the socket portion 123.

The movement of the nut retaining member 124 when the nut 202 is held in the nut retaining member 124 of the end effector 12, and the movement of the nut retaining member 124 when the collar 202b is removed from the nut retaining member 124 of the end effector 12, will be explained below with reference to Figures 4 and 5.

As shown in Figure 4(a), when the collar 202b of the nut 202, handled by an operator (skilled worker) using the assembly system 100, is pressed against the claw portion 127b from one side Da1 and against the elastic force of the elastic member 128, the second portion 127 and the first portion 126 of the nut holding member 124 rotate together with the pivot shaft 125. As the second portion 127 and the first portion 126 rotate together with the pivot shaft 125, the gap between the claw portions 127b widens to a degree that allows the nut 202 to pass through. Then, as shown in Figure 4(b), the collar 202b of the nut 202 that has passed between the claw portions 127b enters the housing portion 123r and is guided to the other side Da2 by the tapered surface 123a, and is positioned in the area defined by the fastening surface 123b and the positioning surface 123c. In this case, since the elastic member 128 is pressing the first portion 126 against one side Da1, the claw portion 127b, which is integrated with the extended portion 127a extending from the first portion 126, comes into contact with the side surface of the nut body 202a. In other words, when the collar 202b is housed in the housing portion 123r, the nut body 202a is gripped (restrained) by the two claw portions 127b. Alternatively, instead of the operator's hand, a jig or actuator may be used to hold the nut 202 in the nut holding member 124 of the end effector 12.

As shown in Figure 5(a), when the nut 202 is fastened to the bolt 201 by the rotation of the socket portion 123 of the nut runner 121, the collar 202b detaches (breaks) from the nut body 202a while only the nut body 202a of the nut 202 is fastened to the bolt 201. The collar 202b, separated from the nut body 202a, is then held by the claw portion 127b of the nut holding member 124. In other words, the nut 202 is collar-cut. Then, as shown in Figure 5(b), when the first portion 126 is pressed against the other side Da2 by the hand of an operator or the like, against the elastic force of the elastic member 128, the first portion 126 and the second portion 127 rotate around the pivot axis 125, and as a result the gap between the claw portions 127b widens to a degree that the collar 202b can pass through. As a result, the collar 202b, which was held by the claw portions 127b, falls out through the gap between the claw portions 127b. In other words, the first part 126 functions as a lever to drop the collar 202b, which is held by the nut holding member 124 after the nut body 202a and bolt 201 have been fastened together, through the gap between the claw portions 127b.

(Control device configuration)
The control device 30 controls the joint angles of the manipulator 10 and the fastening motor 120a of the main body 120 of the end effector 12 so that the nut 202 held by the end effector 12 is fastened to the bolt 201 provided on the object to be worked on 200. As shown in Figure 7, the control device 30 includes, for example, an acquisition unit 31, a calculation unit 32, a coordinate transformation unit 33, a correction unit 34, a determination unit 35, a control unit 36, a learning unit 37, a counter unit 38, and a storage unit 39.

(Acquisition Department)
The acquisition unit 31 acquires sensor data over time by receiving the data transmitted from the encoder 130 (first sensor 13) and the force sensor 140 (second sensor 14) in real time. The acquisition unit 31 acquires the joint angles of each joint 110 (first joint 111 to sixth joint 116) from the encoder 130 at a first predetermined period, and acquires force data from the force sensor 140 at a first predetermined period. The first predetermined period is determined, for example, based on the sampling rate of the encoder 130 or the force sensor 140. The acquisition unit 31 also acquires images over time by receiving the data transmitted from the imaging device 20 in real time. The acquisition unit 31 acquires images from the imaging device 20 at a second predetermined period. The second predetermined period is determined, for example, based on the frame rate (fps: frames per second) of the imaging device 20. The second predetermined period may be the same as or different from the first predetermined period. Furthermore, the acquisition unit 31 acquires the center position coordinates of the bolt 201 that is the target of the most recent construction from the CAD data 395 (Computer-Aided Design) of the construction target 200, which is stored in advance by the storage unit 39. The CAD data 395 of the construction target 200 includes, for example, point cloud data that shows the contour of the construction target 200 in three dimensions. Each point that makes up the point cloud data is assigned three-dimensional coordinates (Xw, Yw, Zw) that are expressed in a world coordinate system that conforms to the origin (0,0,0) of the manipulator 10.

The acquisition unit 31 sends the acquired joint angles of each joint 110 to the calculation unit 32 and stores them in the storage unit 39. It also sends the acquired force data to the correction unit 34 and stores it in the storage unit 39. The acquired center position coordinates of the bolts 201 are sent to the coordinate transformation unit 33 and stored in the storage unit 39. The acquired images are sent to the determination unit 35 and the control unit 36 and stored in the storage unit 39. Furthermore, when the acquisition unit 31 sends the acquired center position coordinates of the bolts 201 to the coordinate transformation unit 33, it sends an acquisition flag to the counter unit 38 indicating that the center position coordinates of the bolt 201 targeted for the most recent construction have been acquired. The acquisition flag is, for example, 1. The counter unit 38 sequentially increments the acquisition flag each time it receives one from the acquisition unit 31, and sends the result of the acquisition flag increment to the sixth determination unit 356.

(Calculation section)
The calculation unit 32 calculates the position coordinates and orientation angles of the nut 202 based on the joint angles of each joint 110 (first joint 111 to sixth joint 116) received from the acquisition unit 31. Specifically, as shown in Figure 8, the calculation unit 32 calculates the position coordinates (Xn, Yn, Zn) of the center portion of the end face 202e of the nut 202, which is relative to the origin of the manipulator 10, and the orientation angles (θx, θy, θz), which are the inclination angles with respect to each of the three coordinate planes (XY plane, YZ plane, ZX plane) of the world coordinate system in which the position coordinates are represented, based on the joint angles of each joint 110. In Figure 8, for illustrative purposes only, only the position coordinates (Xn, Yn, Zn) of the nut 202 are shown. The origin of the manipulator 10 may be any point set within the manipulator 10, or it may be any point set within the object 200 after calibration has been performed to adjust the positional relationship between the manipulator 10 and the object 200. Figure 8 shows an example where the origin of the manipulator 10 is located within the object 200.

The calculation unit 32 sends the calculated position coordinates of the nut 202 to the coordinate transformation unit 33 and stores them in the storage unit 39. It also sends the calculated attitude angle of the nut 202 to the determination unit 35 and the control unit 36 and stores it in the storage unit 39. Alternatively, the calculation unit 32 may obtain the position coordinates and attitude angle of the nut 202 by inputting the joint angles of each joint 110 into a pre-stored correspondence information (such as a function) in the storage unit 39, which associates the joint angles of each joint 110 with the position coordinates and attitude angle of the nut 202.

(Coordinate transformation section)
The coordinate transformation unit 33 converts the position coordinates of the nut 202 received from the calculation unit 32 into position coordinates expressed in the bolt center coordinate system, based on the center position coordinates of the bolt 201 received from the acquisition unit 31. Specifically, the coordinate transformation unit 33 converts the position coordinates of the nut 202 into the bolt center coordinate system (Xb, Yb, Zb) as shown by arrow A (vector) in Figure 8 by subtracting a vector indicating the center position coordinates of the bolt 201 to be installed, received from a vector indicating the position coordinates of the nut 202 expressed in the world coordinate system. In this embodiment, the center position coordinates of the bolt 201 (origin of the bolt center coordinate system) are located somewhere on the center line Ar2 of the bolt 201, for example, at a point where the surface of the object to be installed 200 and the center line Ar2 of the bolt 201 virtually intersect. The coordinate transformation unit 33 sends the position coordinates of the nut 202 converted to the bolt center coordinate system to the determination unit 35 and the control unit 36, and stores them in the storage unit 39.

(Correction section)
The correction unit 34 corrects the contact force included in the force data received from the acquisition unit 31. Figure 9 is a diagram for conceptually explaining the contact force that the nut 202 receives from the bolt 201 when the nut 202 held by the end effector 12 comes into contact with the bolt 201. In Figure 9, for convenience of illustration, the contact force acting in the axial direction Da is shown by arrow Fb, and the torque acting around the rotation axis Ar is shown by arrow Ft. Figure 10 is a diagram for conceptually explaining the correction value (arrow Fg1) used by the correction unit 34 to correct the contact force (arrow Fb). In Figure 10, Dv indicates the vertical direction, and CG indicates the center of gravity of the end effector 12. The correction unit 34 corrects the contact force (arrow Fb) received from the acquisition unit 31 by adding a correction value (arrow Fg1) based on the weight of the end effector 12 (arrow Fg) to the contact force. The correction value (arrow Fg1) is negative when, for example, the weight of the end effector 12 due to gravity (arrow Fg) has a directional component on one side Da1 in the axial direction Da (Figure 10 (a)). On the other hand, the correction value (arrow Fg1) is positive when, for example, the weight of the end effector 12 (arrow Fg) has a directional component on the other side Da2 in the axial direction Da (Figure 10 (b)). The correction value is pre-stored in the storage unit 39 as correspondence relationship information (such as a function) associated with the joint angle of each joint 110. The correction unit 34 obtains the correction value by referring to the correspondence relationship information stored in the storage unit 39 in a timely manner. The correction unit 34 sends the force data including the corrected contact force to the determination unit 35 and the control unit 36, and stores it in the storage unit 39.

(Judgment Department)
The determination unit 35 performs various determination processes based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate transformation unit 33, and the force data received from the correction unit 34. Returning to Figure 7, the determination unit 35 includes, for example, a first determination unit 351, a second determination unit 352, a third determination unit 353, a fourth determination unit 354, a fifth determination unit 355, and a sixth determination unit 356.

(First judgment part)
The first determination unit 351 determines whether or not the nut 202 is in contact with the bolt 201 based on the force data received from the correction unit 34. Hereinafter, the determination made by the first determination unit 351 will be referred to as the "first determination". For example, the first determination unit 351 determines that the nut 202 is not in contact with the bolt 201 if the contact force included in the received force data is less than a first threshold value indicating the magnitude of a predetermined contact force. On the other hand, the first determination unit 351 determines that the nut 202 is in contact with the bolt 201 if the contact force included in the acquired force data is equal to or greater than the first threshold value. Hereinafter, the position of the nut 202 when it is not in contact with the bolt 201 will be referred to as the "first position", and the position of the nut 202 when it is in contact with the bolt 201 will be referred to as the "second position". The state in which the nut 202 and bolt 201 are screwed together, as described later, is included in the state in which the nut 202 is in the second position. The first threshold value is stored in advance, for example, in the memory unit 39. The first determination unit 351 performs the first determination by referring to the first threshold value stored in the memory unit 39 in a timely manner. The first determination unit 351 sends the result of the first determination, indicating whether or not the nut 202 is in contact with the bolt 201, to the control unit 36.

(Second judgment section)
The second determination unit 352 determines whether there is an abnormality in the state of the nut 202 based on the attitude angle of the nut 202 received from the calculation unit 32 and the position coordinates of the nut 202 received from the coordinate transformation unit 33. Hereinafter, the determination by the second determination unit 352 will be referred to as the "second determination". The second determination unit 352 determines that there is a problem with the position coordinates of the nut 202 if the received position coordinates of the nut 202 deviate from a predetermined coordinate region (three-dimensional region) expressed in the bolt center coordinate system. On the other hand, the second determination unit 352 determines that there is no problem with the position coordinates of the nut 202 if the received position coordinates of the nut 202 are within the coordinate region. Furthermore, the second determination unit 352 determines that there is a problem with the attitude angle of the nut 202 if the received attitude angle of the nut 202 is greater than or equal to a predetermined attitude angle threshold. On the other hand, the second determination unit 352 determines that there is no problem with the attitude angle of the nut 202 if the received attitude angle of the nut 202 is less than the attitude angle threshold. The second determination unit 352 determines that there is an abnormality in the condition of the nut 202 if it determines that there is a problem with one or more of the position coordinates and attitude angles of the nut 202. On the other hand, the second determination unit 352 determines that there is no abnormality in the condition of the nut 202 if it determines that there is no problem with either the position coordinates or attitude angles of the nut 202. The above coordinate region and attitude angle threshold values are stored in advance in, for example, the storage unit 39. The second determination unit 352 performs the above second determination by referring to the coordinate region and attitude angle threshold values stored in the storage unit 39 in a timely manner. The second determination unit 352 sends the result of the second determination to the control unit 36.

(Third judgment section)
The third determination unit 353 determines whether the screwing of the nut 202 was successful based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate transformation unit 33, and the force data received from the correction unit 34. In other words, the third determination unit 353 determines whether the screwing of the nut 202 and the bolt 201 is successful or not. In this specification, "screwing" means, for example, the state in which the first thread closest to the end face 202e in the nut hole 202h, which is the female thread portion of the nut 202, is engaged with the first thread closest to the end face 201e in the exposed male thread portion 201a of the bolt 201. In this embodiment, when the third determination unit 353 receives a dataset including an image, the position coordinates and attitude angle of the nut 202, and force data, it performs a determination using a trained model that has been trained to output a determination result indicating whether the screwing of the nut 202 and the bolt 201 is successful or not. Hereinafter, the determination made by the third determination unit 353 will be referred to as the "third determination," and the trained model used by the third determination unit 353 for the third determination will be referred to as the "third trained model 393." That is, the third trained model 393 takes a dataset including an image, the position coordinates and orientation angle of the nut 202, and force data as input elements, and outputs a determination result indicating whether the nut 202 and bolt 201 are screwed together or not as output elements. The third trained model 393 is stored in advance, for example, in the memory unit 39. The third determination unit 353 obtains the determination result output by inputting the above dataset into the third trained model 393 stored in the memory unit 39 as the result of the third determination.

The third pre-trained model 393 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN). The third pre-trained model 393 is generated (learned) through a learning step that is repeated multiple times (e.g., several hundred times) in which the above dataset is input and the model is taught whether the nut 202 was successfully screwed onto the input dataset (ground truth data accurately judged by a human as being screwed on or not screwed on). Note that the third pre-trained model 393 may use, for example, a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

The third determination unit 353 outputs a flag (i=1) indicating that a retry operation (described later) is necessary if the result of the third determination indicates that the screwing of the nut 202 and bolt 201 has failed. On the other hand, the third determination unit 353 outputs a flag (i=0) indicating that a retry operation is not necessary if the result of the third determination indicates that the screwing of the nut 202 and bolt 201 has been successful. Hereinafter, the flag (i=1 or 0) output by the third determination unit 353 regarding the necessity of a retry operation will be referred to as the "retry flag". That is, the retry flag output by the third determination unit 353 indicates the result of the third determination and has a value of 1 or 0. The third determination unit 353 sends the retry flag, which is the result of the third determination, to the control unit 36 and the counter unit 38 and stores it in the storage unit 39. Furthermore, the counter unit 38 sequentially increments the retry flag each time it receives a retry flag from the third determination unit 353, and sends the result of the retry flag increment to the sixth determination unit 356.

(Fourth judgment part)
The fourth determination unit 354 determines whether the fastening of the nut 202 was successful based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate transformation unit 33, and the force data received from the correction unit 34. In other words, the fourth determination unit 354 determines whether the fastening of the nut 202 and the bolt 201 was successful or not. In this specification, "fastening" means, for example, the state in which the last thread furthest from the end face 202e in the nut hole 202h, which is the female thread portion of the nut 202, is fitted into the last thread furthest from the end face 201e in the exposed male thread portion 201a of the bolt 201. In this embodiment, when the fourth determination unit 354 receives a dataset including an image, the position coordinates and attitude angle of the nut 202, and force data, it performs a determination using a trained model that has been trained to output a determination result indicating whether the fastening of the nut 202 and the bolt 201 was successful or not. Hereinafter, the determination made by the fourth determination unit 354 will be referred to as the "fourth determination," and the trained model used by the fourth determination unit 354 for the fourth determination will be referred to as the "fourth trained model 394." That is, the fourth trained model 394 takes a dataset including an image, the position coordinates and orientation angle of the nut 202, and force data as input elements, and outputs a determination result indicating whether the fastening of the nut 202 and bolt 201 is successful or not as output elements. The fourth trained model 394 is pre-stored in, for example, the memory unit 39. The fourth determination unit 354 obtains the determination result output by inputting the above dataset into the fourth trained model 394 stored in the memory unit 39 as the result of the fourth determination.

The fourth pre-trained model 394 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN). The fourth pre-trained model 394 is generated (learned) by repeating a learning step multiple times (e.g., several hundred times) in which the above dataset is input and the model is taught whether the nuts 202 were fastened or not (ground truth data accurately judged by a human as being fastened or not fastened). Note that the fourth pre-trained model 394 may use, for example, a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

The fourth determination unit 354 outputs a flag (i=1) indicating that a retry operation is necessary if the result of the fourth determination indicates that the fastening of the nut 202 and bolt 201 has failed. On the other hand, the fourth determination unit 354 outputs a flag (i=0) indicating that a retry operation is not necessary if the result of the fourth determination indicates that the fastening of the nut 202 and bolt 201 has been successful. Hereinafter, the flag (i=1 or 0) related to the necessity of a retry operation output by the fourth determination unit 354 will be referred to as the "retry flag" without distinction from the retry flag output by the third determination unit 353. That is, the retry flag output by the fourth determination unit 354 indicates the result of the fourth determination and has a value of 1 or 0. The fourth determination unit 354 sends the retry flag, which is the result of the fourth determination, to the fifth determination unit 355, the control unit 36, and the counter unit 38, and stores it in the storage unit 39. Furthermore, the counter unit 38 sequentially increments the retry flag each time it receives a retry flag from the fourth determination unit 354, and sends the result of the retry flag increment to the sixth determination unit 356.

(Fifth judgment part)
The fifth determination unit 355 determines whether the collar 202b has broken and been cut (collar cut) when the result of the fourth determination received from the fourth determination unit 354 indicates that the fastening of the nut 202 has been successful (i = 0). Specifically, the fifth determination unit 355 determines whether the collar 202b of the nut 202 has separated from the nut body 202a, for example, based on force data received from the correction unit 34. Hereinafter, the determination made by the fifth determination unit 355 will be referred to as the "fifth determination". For example, the fifth determination unit 355 determines that the collar 202b has been cut if the contact force included in the received force data is less than a second threshold value indicating a predetermined magnitude of contact force. On the other hand, the fifth determination unit 355 determines that the collar 202b has not been cut if the contact force included in the acquired force data is equal to or greater than the second threshold value.

The second threshold value is, for example, pre-stored in the memory unit 39. The fifth determination unit 355 performs the fifth determination by referring to the second threshold value stored in the memory unit 39 in a timely manner. Alternatively, instead of the above determination operation, the fifth determination unit 355 may determine whether the fastening of the nut 202 was successful based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate transformation unit 33, and the force data received from the correction unit 34. In this case, the fifth determination unit 355 may perform the determination using a fifth trained model 396, which has been trained to output a determination result indicating whether the color 202b has been cut when a dataset including the image, the position coordinates and attitude angle of the nut 202, and the force data is input. The fifth trained model 396 is, for example, pre-stored in the memory unit 39. The fifth determination unit 355 sends the result of the fifth determination to the control unit 36 and stores it in the memory unit 39.

(6th judgment section)
The sixth determination unit 356 makes a predetermined determination based on the summation result of various flags received from the counter unit 38. Hereinafter, the determination made by the sixth determination unit 356 will be referred to as the "sixth determination". The sixth determination unit 356 determines whether or not nuts 202 have been fastened to all bolts 201 based on the summation result of acquired flags received from the counter unit 38. Specifically, the sixth determination unit 356 determines that nuts 202 have been fastened to all bolts 201 if the summation result of acquired flags received from the counter unit 38 is equal to the number of bolts 201 that are the target of construction and are provided on the object 200 to be constructed. On the other hand, the sixth determination unit 356 determines that nuts 202 have not been fastened to all bolts 201 if the summation result of acquired flags received from the counter unit 38 is less than the number of bolts 201 provided on the object 200 to be constructed. The number of bolts 201 installed on the object to be constructed 200 is pre-stored, for example, in CAD data 395 in the memory unit 39, and is an integer (for example, several thousand to tens of thousands).

Furthermore, the sixth determination unit 356 determines whether a predetermined number of retries have been performed based on the result of adding the retry flag received from the counter unit 38. Specifically, the sixth determination unit 356 determines that a predetermined number of retries have been performed if the result of adding the retry flag received from the counter unit 38 is equal to or greater than a predetermined retry threshold. On the other hand, the sixth determination unit 356 determines that a predetermined number of retries have not been performed if the result of adding the retry flag received from the counter unit 38 is less than the retry threshold. The retry threshold is, for example, stored in advance in the storage unit 39 and is an integer. The sixth determination unit 356 performs the sixth determination by referring to the retry threshold stored in the storage unit 39 in a timely manner. The sixth determination unit 356 sends the result of the sixth determination to the learning unit 37 and the control unit 36.

(Control Unit)
The control unit 36 controls the movement of the end effector 12 based on the results of various determinations received from the determination unit 35. The control unit 36 includes, for example, a first control unit 361, a second control unit 362, and a third control unit 363.

(First Control Unit)
If the result of the first determination indicates that the nut 202 is in contact with the bolt 201 (i.e., the nut 202 is in the second position), the first control unit 361 executes a fastening operation of the nut 202 to the bolt 201 based on a dataset that includes the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate transformation unit 33, and the force data received from the correction unit 34. The "fastening operation" here includes changing the position coordinates and attitude angle of the nut 202 so that the nut 202 and the bolt 201 are screwed together when the nut 202 is in the second position, and rotating the nut 202 so that the nut 202 and the bolt 201 are fastened together when the nut 202 and the bolt 201 are screwed together (i.e., the nut 202 is in the second position). Specifically, the first control unit 361 controls the movement and rotation of the end effector 12 using a learned model. Here, "movement of the end effector 12" means that, for example, the end effector 12 and the nut 202 held by the end effector 12 move as a result of controlling the joint angles of the motors 117 of each joint portion 110 of the arm 11, and "rotation of the end effector 12" means that, for example, the nut 202 rotates as a result of controlling the fastening motor 120a of the main body portion 120 of the end effector 12. Hereinafter, the learned model used by the first control unit 361 for the above control of the end effector 12 will be referred to as the "first learned model 391". The first learned model 391 is stored in advance in the memory unit 39, for example.

The first trained model 391 is trained to output command values indicating the position coordinates and attitude angle of the nut 202 at a second time point, which is after the first time point and approaches the target position coordinates and attitude angle of the nut 202 indicating that the nut 202 is fastened to the bolt 201, when a dataset including the position coordinates and attitude angle of the nut 202 at the first time point and the contact force and torque at the first time point is input, and the command values indicating the position coordinates and attitude angle of the nut 202 at the second time point are output. The time interval between the first time point and the second time point is, for example, the first predetermined period described above. The target position coordinates and target orientation angle of the nut 202 refer to the position coordinates and orientation angle of the nut 202 that indicate the state in which the female thread portion of the nut 202 and the male thread portion 201a of the bolt 201 to be fastened are precisely engaged with each other and fastening is completed. In this embodiment, the first control unit 361 controls the joint angles of the motors 117 of each joint portion 110 and the fastening motor 120a of the end effector 12 of the controlled device S, based on the command values output by inputting the above dataset into the first trained model 391 stored in the memory unit 39, thereby controlling the position coordinates and orientation angle of the nut 202. The first trained model 391 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN). Note that the first trained model 391 may be a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN). Furthermore, the first control unit 361 sequentially stores the joint angles of each joint 110, which have been changed over time, and the rotational speed of the fastening motor 120a of the end effector 12 (the screw feed rate associated with the rotation of the nut 202) in the storage unit 39 at each time step.

(Second Control Unit)
The second control unit 362 controls the movement and rotation of the end effector 12 based on a dataset that includes an image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, and the position coordinates of the nut 202 received from the coordinate transformation unit 33. Specifically, the second control unit 362 uses a trained model to control the movement and rotation of the end effector 12, moving the nut 202 from a first position where it is not in contact with the bolt 201 to a second position where it is in contact with the bolt 201. In other words, the second control unit 362 causes the nut 202 in the first position to approach the bolt 201 that is the target of installation from the outside. Hereinafter, the trained model used by the second control unit 362 to control the end effector 12 will be referred to as the "second trained model 392". The second trained model 392 is stored in advance, for example, the storage unit 39.

The second pre-trained model 392 is trained to output command values indicating the position coordinates and attitude angle of the nut 202 at a fourth time step, which occurs after the third time step and when the nut 202 has approached the second position, when a dataset including an image acquired at the first position at the third time step and the position coordinates and attitude angle of the nut 202 at the third time step is input. In other words, the second pre-trained model 392 takes a dataset including an image acquired at the first position at the third time step and the position coordinates and attitude angle of the nut 202 at the third time step as input elements, and outputs command values indicating the position coordinates and attitude angle of the nut 202 at the fourth time step. The time interval between the third time step and the fourth time step is, for example, the second predetermined period described above. In this embodiment, the second control unit 362 controls the joint angle of the motor 117 of each joint 110 of the controlled device S based on the command value output by inputting the above dataset into the second trained model 392 stored in the memory unit 39, thereby bringing the nut 202 closer to the bolt 201 (making it approach). The second trained model 392 is, for example, a deep learning model (supervised learning model) such as a deep neural network (DNN). Note that the second trained model 392 may also use deep learning models such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

(Third Control Unit)
The third control unit 363 executes a retry operation when it receives a result of a third determination (i=1) indicating that the screwing of the nut 202 and bolt 201 has failed, or when it receives a result of a fourth determination (i=1) indicating that the fastening of the nut 202 and bolt 201 has failed. Specifically, the third control unit 363 executes the control of the end effector 12 (fastening operation steps S106, S4, etc., described later) in reverse order, from a predetermined time in the past to the time when the result of the third or fourth determination was received. The "predetermined time in the past" here means, for example, the time when the control of the end effector 12 was started (the time when the fastening operation steps S106, S4, etc., described later were started). The third control unit 363 controls the movement and rotation of the end effector 12 in reverse order, while referring in a timely manner to the time-series data of the joint angles of each joint 110 stored in the memory unit 39, and the rotational speed of the fastening motor 120a of the end effector 12 (screw feed rate accompanying the rotation of the nut 202).

(Learning Department)
When the learning unit 37 receives the result of the sixth determination from the sixth determination unit 356, which indicates that nuts 202 have been fastened to all bolts 201, it modifies the parameter weight values in the first trained model 391 based on various data stored in the storage unit 39. The learning unit 37 modifies (trains) the parameter weight values in the first trained model 391 using, for example, backpropagation. When the learning unit 37 modifies the parameters of the first trained model 391, it sends a modification flag to the counter unit 38 to indicate that a modification has been made. The modification flag is, for example, 1.

The counter unit 38 sequentially increments the correction flag received from the learning unit 37 each time it receives one, and sends (returns) the result of the correction flag increment to the sixth determination unit 356. The sixth determination unit 356 compares the result of the correction flag increment received from the counter unit 38 with a predetermined value. If the result of the correction flag increment is greater than or equal to the predetermined value, the sixth determination unit 356 determines that the correction has been made a predetermined number of times. On the other hand, if the result of the correction flag increment is less than the predetermined value, the sixth determination unit 356 determines that the correction has not been made a predetermined number of times. The predetermined value is, for example, an integer, pre-stored in the storage unit 39. The sixth determination unit 356 performs this sixth determination by referencing the predetermined value stored in the storage unit 39 in a timely manner.

(Method for teaching the first pre-trained model)
The learning steps (how the model is taught) of the first trained model 391 used by the first control unit 361 will be described below with reference to Figure 11. Figure 11 is a flowchart showing an example of the learning steps of the first trained model 391. For the sake of explanation, the model will be referred to as a "trained model" without distinction, even if it is a model that has not yet been trained.

First, the acquisition unit 31 acquires data for the bolt 201 that is the target of the most recent installation from CAD data 395 or the like that is pre-stored in the storage unit 39 (step S101). Next, the acquisition unit 31 acquires the known position coordinates and known orientation angle of the nut 202 (step S102). The "known position coordinates and known orientation angle" of the nut 202 acquired by the acquisition unit 31 here is acquired, for example, from a variation generation model 397 (see Figure 7) that is pre-stored in the storage unit 39. When the variation generation model 397 is referenced by the acquisition unit 31, it varies the arbitrary position coordinates (second position) and orientation angle of the end face 201e of the bolt 201 according to a probability distribution such as a normal distribution (a state in which errors are deliberately given randomly), and outputs it to the acquisition unit 31. The position coordinates averaged by the variation generation model 397 are, for example, the center position coordinates of the end face 201e of the bolt 201, and the attitude angle averaged by the variation generation model 397 is, for example, the attitude angle when the rotation axis Ar and the center line Ar2 of the bolt 201 coincide in a straight line. The center position coordinates of the end face 201e of the bolt 201 are expressed, for example, in the bolt center coordinate system (Xb, Yb, Zb). In other words, the acquisition unit 31 acquires the known position coordinates and known attitude angle of the nut 202 by referring to the variation generation model 397. Next, the second control unit 362 uses the second trained model 392 to bring the nut 202 to approach the target bolt 201 (step S103). Note that in step S103, it is not necessary to use the second trained model 392 to bring the nut 202 to approach the bolt 201. In this case, the control unit 36 may, for example, guide the nut 202 towards the target bolt 201 according to a predetermined approach operation pre-stored in the memory unit 39. Next, the first determination unit 351 performs a first determination to determine whether the nut 202 has made contact with the bolt 201 (step S104). If the result of the first determination indicates that the nut 202 has made contact with the bolt 201 (step S104: YES), the position coordinates and orientation angle of the nut 202 are corrected to the known position coordinates and orientation angle of the nut 202 obtained in step S102, either by control by the control unit 36 or manually by the operator (step S105). On the other hand, if the result of the first determination indicates that the nut 202 has not made contact with the bolt 201 (step S104: NO), the process returns to step S103. Following step S105, the fastening operation step S106 is executed.

In the fastening operation step S106, the control unit 36 starts the fastening operation of the nut 202 according to a predetermined procedure. In this embodiment, the fastening operation of the nut 202 according to a predetermined procedure is referred to as the "teaching operation". The "predetermined procedure" here refers to data that shows command values indicating the position coordinates and attitude angle of the nut 202 in a time series, for changing the known position coordinates (second position) and known attitude angle of the nut 202 to the target position coordinates and target attitude angle of the nut 202. This data is stored in advance in the storage unit 39, for example. The predetermined procedure may be, for example, correspondence information (such as a function) in which the known position coordinates and known attitude angle of the nut 202 at the start time of the fastening operation, multiple times after the start time of the fastening operation, and the command values of the position coordinates and attitude angle of the nut 202 corresponding to each time are related to each other. The control unit 36 controls the movement and rotation of the end effector 12 according to the predetermined procedure. During the teaching operation in which the control unit 36 controls the end effector 12, the acquisition unit 31, calculation unit 32, coordinate transformation unit 33, and correction unit 34 sequentially store the processing results in the storage unit 39. The processing results of the acquisition unit 31, calculation unit 32, coordinate transformation unit 33, and correction unit 34 stored in the storage unit 39 during the teaching operation are used as the training dataset for training the first trained model 391. The fastening operation step S106 ends when the control unit 36 finishes controlling the end effector 12 based on the final command value included in a predetermined procedure.

Following the fastening operation step S106, the fourth determination unit 354 performs a fourth determination to determine whether the fastening of the nut 202 was successful (step S107). If the result of the fourth determination indicates that the fastening of the nut 202 was successful (step S107: YES), the sixth determination unit 356 determines whether the nut 202 was fastened to all bolts 201 (step S108). On the other hand, if the result of the fourth determination indicates that the fastening of the nut 202 failed (step S107: NO), the retry operation step S111 is executed. In the retry operation step S111, the third control unit 363 performs the retry operation described above. When the retry operation is completed, the process returns to step S105.

Now, returning to the explanation of the determination process in step S108, if the result of the sixth determination indicates that nuts 202 have been fastened to all bolts 201 (step S108: YES), the learning unit 37 modifies the weight values of the parameters of the first trained model 391 based on the various data acquired in the fastening operation step S106 (step S109). On the other hand, if the result of the sixth determination indicates that nuts 202 have not been fastened to all bolts 201 (step S108: NO), the process proceeds to step S101. Following the process in step S109, the sixth determination unit 356 determines whether the learning unit 37 has modified the weight values of the parameters of the first trained model 391 a predetermined number of times (step S110). If the result of the determination by the sixth determination unit 356 indicates that the modification has been made a predetermined number of times (step S110: YES), the learning step is completed. On the other hand, if the result of the determination by the sixth determination unit 356 indicates that the predetermined number of corrections have not been made (step S110: NO), the process returns to step S109.

Based on the various data acquired through the learning steps described above, the learning unit 37 repeatedly modifies the parameter weight values, thereby teaching the first trained model 391 to output command values corresponding to the input. Specifically, the first trained model 391 is learned to output command values corresponding to the input through repeated learning steps in which a training dataset is input, which includes the position coordinates and attitude angles of multiple nuts 202 acquired during a teaching operation that controls the end effector 12 so that the known position coordinates and attitude angles of the nuts 202, which are randomly deviated by a given amount from the target position coordinates and target attitude angles of the nuts 202, become the target position coordinates and target attitude angles, as well as the contact force and torque corresponding to the time when each nut 202's position coordinates and attitude angle were acquired.

(Second method of teaching the trained model)
Furthermore, the second trained model 392 used by the second control unit 362 is trained (reverse-taught) to output command values corresponding to the input by repeatedly performing a learning step in which a training dataset is input, which includes multiple images acquired during a direct teaching operation to move the end effector 12 from the first position to the second position, and the position coordinates and attitude angle of the nut 202 corresponding to the time when each image was acquired. Specifically, in this embodiment, reverse teaching involves transforming the time series of the image acquired by the acquisition unit 31 when the end effector 12 is moved from the second position to the first position (direct teaching operation) and the time series of the position coordinates and attitude angle of the nut 202, so that the time series is reversed, and training the second trained model 392 with the transformed time series dataset as the training dataset. Also, "direct teaching operation" as used here refers to the operation of moving the end effector 12 so that the nut 202 moves from the second position to the first position, such as by manual operation by an operator, as shown as an example in Figure 12. The above learning steps for training (reverse-teaching) the second pre-trained model 392 are repeated, for example, several dozen to several hundred times.

(Operation of the control device)
Next, an example of the operation of the control device 30 in this embodiment will be described with reference to Figure 13. However, the order of the processes described below is not limited to the following example and may be rearranged as appropriate.

First, the second control unit 362 uses the second learned model 392 to bring the nut 202 close to the bolt 201 to be installed (step S1). Next, the second determination unit 352 performs a second determination to determine whether or not there is an abnormality in the state of the nut 202 (step S2). If the result of the second determination indicates that there is an abnormality in the state of the nut 202 (step S2: YES), for example, the control unit 36 operates to move the nut 202 away from the bolt 201, and then the process returns to step S1. On the other hand, if the result of the second determination indicates that there is no abnormality in the state of the nut 202 (step S2: NO), the first determination unit 351 performs a first determination to determine whether or not the nut 202 has come into contact with the bolt 201 (step S3). If the result of the first determination indicates that the nut 202 has not come into contact with the bolt 201 (step S3: NO), for example, the control unit 36 operates to move the nut 202 away from the bolt 201, and then the process returns to step S1. On the other hand, if the result of the first determination indicates that the nut 202 has made contact with the bolt 201 (step S3: YES), the fastening operation step S4 is executed. In the fastening operation step S4, the first control unit 361 executes the fastening operation of the nut 202 to the target bolt 201 by controlling the movement and rotation of the end effector 12 using the first learned model 391. In the fastening operation step S4, while the first control unit 361 is executing the fastening operation, the third determination unit 353 may perform a third determination to determine whether the screwing of the nut 202 was successful or not. If the result of the third determination indicates that the screwing of the nut 202 was successful, the first control unit 361 continues to execute the fastening operation. On the other hand, if the result of the third determination indicates that the screwing of the nut 202 failed, the first control unit 361 stops the fastening operation. If the first control unit 361 stops the fastening operation, for example, the retry operation step S8 is executed.

When the fastening operation by the first control unit 361 is completed, the fourth determination unit 354 performs a fourth determination to determine whether or not the fastening of the nut 202 was successful (step S5). If the result of the fourth determination indicates that the fastening of the nut 202 was successful (step S5: YES), the fifth determination unit 355 determines whether or not the collar 202b has broken and been cut (step S6). On the other hand, if the result of the fourth determination indicates that the fastening of the nut 202 was unsuccessful (step S5: NO), the retry operation step S8 is executed. In the retry operation step S8, the third control unit 363 performs the retry operation described above. When the retry operation step S8 is completed, the sixth determination unit 356 performs a sixth determination to determine whether or not a predetermined number of retries have been performed (step S9). If the result of the sixth determination indicates that a predetermined number of retries have been performed (step S9: YES), for example, the control device 30 terminates its operation. On the other hand, if the result of the sixth determination indicates that a predetermined number of retries have not been performed (step S9: NO), the process proceeds to step S1. Now, returning to the explanation of the determination process in step S6, if the result of the fifth determination indicates that the collar 202b has not been cut (step S6: NO), the process returns to the fastening operation step S4. On the other hand, if the result of the fifth determination indicates that the collar 202b has been cut (step S6: YES), the sixth determination unit 356 determines whether or not nuts 202 have been fastened to all bolts 201 (step S7). If the result of the sixth determination indicates that nuts 202 have not been fastened to all bolts 201 (step S7: NO), the process proceeds to step S1. On the other hand, if the result of the sixth determination indicates that nuts 202 have been fastened to all bolts 201 (step S7: YES), for example, the control device 30 terminates its operation.

The operation of the control device 30 described above is repeatedly performed during the assembly process of the object to be constructed 200.

(Effects/Mechanisms)
In this embodiment, the movement and rotation of the end effector 12 are controlled based on command values of the position coordinates and orientation angles of the nut 202 at a second time, output by a first learned model 391 that takes as input the position coordinates and orientation angles of the nut 202 based on the joint angles of each joint 110 at a first time, and the contact force and torque transmitted from the end effector 12 at a first time. At this time, the position coordinates and orientation angles of the nut 202 in the command values of the second time output by the first learned model 391 are approaching the target position coordinates and target orientation angles of the nut 202 that indicate the state in which the nut 202 is fastened to the bolt 201. In other words, each time input is made to the first learned model 391, the state of the nut 202 can be transitioned toward the state in which the nut 202 is fastened to the bolt 201. Therefore, the nut 202 can be fastened more smoothly to the bolt 201 provided on the object to be constructed 200.

Furthermore, in this embodiment, the first trained model 391 is pre-trained to perform the above input and output by repeatedly executing a teaching that fastens a nut 202, which has known position coordinates and an attitude angle that deviates from the target position coordinates and target attitude angle, to a bolt 201. Therefore, for example, even if there is a deviation in the position coordinates or attitude angle of the nut 202 relative to the bolt 201 during actual operation, the actual time taken for fastening can be suppressed from deviating from the predicted fastening time. As a result, for example, the man-hours required for assembly can be reduced.

Furthermore, in this embodiment, the movement and rotation of the end effector 12 are controlled based on command values indicating the position coordinates and orientation angle of the nut 202 at a fourth time, output by a second trained model 392 that takes an image of the object to be constructed 200 at a third time, acquired at a first position where the nut 202 is not in contact with the bolt 201, and the position coordinates and orientation angle of the nut 202 at the third time as input. At this time, the position coordinates and orientation angle of the nut 202 in the command values for the fourth time output by the second trained model 392 are approaching the position coordinates and orientation angle of the second position where the nut 202 is in contact with the bolt 201. In other words, each time input is made to the second trained model 392, the nut 202 located at the first position can be moved toward the second position. Therefore, the nut 202 can be smoothly approached by the bolt 201 provided on the object to be constructed 200. Furthermore, by using images as input elements for the second pre-trained model 392, the above effects can be achieved with higher accuracy compared to, for example, a case where images are not used as input elements.

Furthermore, in this embodiment, the second trained model 392 is pre-trained (reverse-taught) to perform the above input and output by repeatedly executing a learning step in which a training dataset containing multiple images acquired during a direct teaching operation to move the end effector 12 from a first position to a second position, and the position coordinates and orientation angle of the nut 202 corresponding to the time each image was acquired, is input. The inventors found that when teaching the second trained model 392 using a direct teaching operation in which an operator using the assembly system 100 manually moves the end effector 12, it is easier to move the end effector 12 from the second position to the first position than to move the nut 202 from the first position to the second position, and that vibrations of the end effector 12 during operation are less likely to occur. As a result, a higher quality training dataset with a lower proportion of noise can be obtained.

Furthermore, in this embodiment, the end effector 12 is controlled using a third trained model 393, which is trained to output a determination result indicating the success or failure of the threading of the nut 202 and bolt 201 when the image acquired at the second position, the position coordinates and orientation angle of the nut 202 corresponding to the time the image was acquired, and the contact force and torque corresponding to the time the image was acquired are input. This prevents, for example, the nut 202 and bolt 201 from proceeding to fasten in an incorrect state. As a result, for example, a decrease in yield during assembly can be suppressed.

Furthermore, in this embodiment, the coordinate system in which the position coordinates of the nut 202 are represented is transformed from the world coordinate system to the bolt-center coordinate system. This allows the relative position coordinates of the nut 202 to the bolt 201 to be represented using the same scale, such as the bolt 201-center coordinates, whereas the position coordinates of the nut 202 expressed in the world coordinate system are unique position coordinates that occur for each bolt 201. As a result, since it is not necessary to teach each bolt 201, generalizability is increased, and the effort and time required to teach the first trained model 391 can be reduced.

Furthermore, in this embodiment, the contact force obtained is corrected using a correction value related to the self-weight of the end effector 12 based on the joint angle of the joint portion 110. This allows the input element of the first trained model 391 to be a contact force closer to the actual contact force than the contact force transmitted from the end effector 12, thereby improving generalizability.

Furthermore, in this embodiment of the end effector 12, the elastic member 128 applies an elastic force to the first portion 126 toward the tip of the socket portion 123, so that the claw portion 127b of the nut holding member 124 grips the nut 202, which has its collar 202b housed in the housing portion 123r of the nut runner 121. This allows the end effector 12 to hold the nut 202 with a simpler configuration compared to, for example, an end effector 12 equipped with a mechanism to hold the nut 202 by suction. Also, the imaging device 20 is positioned inside the socket portion 123 of the nut runner 121. The imaging device 20 irradiates light toward the object to be constructed 200 through the nut hole 202h and generates an image of the object to be constructed 200. That is, the imaging device 20 can generate an image while concentrating the illumination of its imaging range. Therefore, it is less susceptible to the influence of the lighting environment at the construction site.

<Second embodiment of the assembly system>
Next, a second embodiment of the assembly system 100 according to the present disclosure will be described with reference to Figure 14. In the second embodiment described below, components common to the first embodiment described above are denoted by the same reference numerals in the figure and their descriptions are omitted. In the second embodiment, the assembly system 100 does not have a force sensor 140, and instead has a current sensor 141. In this embodiment, the force data described in the first embodiment corresponds to the current value acquired by the current sensor 141, and the force data described above can be read by appropriately replacing it with the current value.

(Current sensor)
The current sensor 141 is a sensor capable of acquiring the current values of the motors 117 (first motor 117a to sixth motor 117f) of each joint 110 (first joint 111 to sixth joint 116). The current sensor 141 is connected to the motors 117 of each joint 110 by wired or wireless communication. The current sensor 141 is an example of the second sensor 14. The current sensor 141 transmits the acquired current values of the motors 117 to the control device 30. The current sensor 141 may be provided on the manipulator 10, for example, or on the control device 30.

(Control device)
In this embodiment, the acquisition unit 31 acquires the current value of the current flowing through each motor 117 from the current sensor 141 at a first predetermined period, sends the acquired current value to the determination unit 35 and the control unit 36, and stores it in the storage unit 39. The first determination unit 351 makes a first determination based on the current value received from the acquisition unit 31. The third determination unit 353 makes a third determination based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate transformation unit 33, and the current value received from the acquisition unit 31. At this time, the third trained model 393 is trained to output a determination result indicating the success or failure of screwing the nut 202 and bolt 201 when a dataset including the image, the position coordinates and attitude angle of the nut 202, and the current value is input. The fourth determination unit 354 performs a fourth determination based on the image received from the acquisition unit 31, the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate transformation unit 33, and the current value received from the acquisition unit 31. At this time, the fourth trained model 394 is trained to output a determination result indicating whether the fastening of the nut 202 and bolt 201 is successful or not when a dataset including the image, the position coordinates and attitude angle of the nut 202, and the current value is input. If the result of the fourth determination received from the fourth determination unit 354 indicates that the fastening of the nut 202 was successful, the fifth determination unit 355 determines whether the collar 202b of the nut 202 is separated from the nut body 202a based on the current value received from the acquisition unit 31. If the first determination result indicates that the nut 202 is in contact with the bolt 201, the first control unit 361 executes a fastening operation of the nut 202 to the bolt 201 based on a dataset that includes the attitude angle of the nut 202 received from the calculation unit 32, the position coordinates of the nut 202 received from the coordinate transformation unit 33, and the current value received from the acquisition unit 31. At this time, the first trained model 391 is trained to output command values that indicate the position coordinates and attitude angle of the nut 202 at a second time point, which is later than the first time point, when a dataset including the position coordinates and attitude angle of the nut 202 at a first time point and the current value at a first time point is input, and the nut 202 has approached the target position coordinates and target attitude angle of the nut 202 indicating that the nut 202 is fastened to the bolt 201.

(Effects/Mechanisms)
In this embodiment, the current values of the motors 117 of each joint 110 acquired by the current sensor 141 can be used to replace the force data described in the first embodiment. That is, the current values of each motor 117 represent the contact force and torque transmitted from the end effector 12 when the nut 202 contacts the bolt 201. As a result, the manipulator 10 does not need to be equipped with a force sensor 140. Therefore, for example, the overall size and weight of the manipulator 10 can be reduced.

<Third embodiment of the assembly system>
Next, a third embodiment of the assembly system 100 according to this disclosure will be described with reference to Figure 15. In the third embodiment described below, components common to the first embodiment described above are denoted by the same reference numerals in the figure and their descriptions are omitted. In the third embodiment, the assembly system 100 further comprises a moving mechanism 16 and a calibration mechanism 17.

(Moving mechanism)
The moving mechanism 16 allows the base portion 15 to move around the workpiece 200. In this embodiment, the moving mechanism 16 is a plurality of wheels 160 provided on the base portion 15. The moving mechanism 16 can travel around the workbench 300. Figure 15 shows an example where four wheels 160 are provided on the base portion 15. The wheels 160 are provided on the base portion 15 so as to be rotatable while in contact with the floor surface. The driving (rotation) of the wheels 160 may be controlled by the control device 30, or it may be done manually by an operator using the assembly system 100.

(Calibration mechanism)
The calibration mechanism 17 determines the relative position between the object to be constructed 200 and the moving mechanism 16. Specifically, the calibration mechanism 17, for example, aligns (calibrates) any point on the object to be constructed 200, represented in a world coordinate system, with any point in the world coordinate system that is based on the origin of the manipulator 10. In other words, the calibration mechanism 17 calibrates the position of the moving mechanism 16 relative to the object to be constructed 200. The calibration mechanism 17 is provided, for example, on the manipulator 10.

(Effects/Mechanisms)
In this embodiment, the base portion 15 is provided with wheels 160 that allow the base portion 15 to move around the object to be constructed 200, so that the manipulator 10 can operate over a wider range. Therefore, for example, when constructing a large object to be constructed 200, it is less likely that it will be necessary to change the position or orientation of the object to be constructed 200.

<Modified form of the third embodiment>
Next, a modified example of the third embodiment of the assembly system 100 according to this disclosure will be described with reference to Figure 16. In this modified example, the configuration of the moving mechanism 16 is different from that of the third embodiment described above.

(Moving mechanism)
In this modified example, the moving mechanism 16 is a linear-acting electric slider 161 positioned between the base portion 15 and the floor surface. The electric slider 161 extends in a long length in one direction. The electric slider 161 is fixed to the floor surface, for example, adjacent to the workbench 300. The electric slider 161 is connected to the base portion 15 while supporting the base portion 15 from below, and moves the base portion 15 in the aforementioned one direction. The drive of the electric slider 161 (linear motion of the base portion 15) may be controlled by the control device 30, or it may be operated manually by an operator. In addition, the electric slider 161 described in this modified example may be positioned separately from the floor surface, for example, mounted on a different moving mechanism 16 than the electric slider 161, instead of being fixed to the floor surface.

(Effects/Mechanisms)
This configuration also produces the same effects and benefits as those described in the third embodiment above.

(Other embodiments)
Although embodiments of this disclosure have been described in detail above with reference to the drawings, the specific configurations are not limited to those of each embodiment, and additions, omissions, substitutions, and other modifications to the configurations are possible without departing from the gist of this disclosure.

Furthermore, the first control unit 361 described in the first embodiment may receive images from the acquisition unit 31. In this case, the first trained model 391 used by the first control unit 361 for control may add the received images to the input elements described above. In this case, in addition to the command value which is the output for the input, the first trained model 391 may generate and output a predicted image at the second time step as a subtask.

Furthermore, the first trained model 391 described in the first embodiment may be trained to generate and output, as a subtask, predicted contact force and torque values at a second time step, in addition to the command value which is the output for the input. The predicted contact force and torque values are values that, when acquired by the acquisition unit 31 at the second time step, represent the contact force and torque predicted by the first trained model 391. These predicted contact force and torque values are not used for control by the first control unit 361. The inventors have found that by having the first trained model 391 output predicted contact force and torque values at a second time step as subtasks, the accuracy of the command value, which is another output element, is improved.

Furthermore, the second trained model 392 used for control by the second control unit 362, as described in the first embodiment, may be trained to generate and output a predicted image at the fourth time step as a subtask, in addition to the command value which is the output for the input. The predicted image is the image predicted by the second trained model 392 when acquired by the acquisition unit 31 at the fourth time step. This predicted image is not used for control by the second control unit 362. The inventors have found that by having the second trained model 392 output predicted values of contact force and torque at the fourth time step as subtasks, the accuracy of the command value, which is another output element, is improved.

Furthermore, the first pre-trained model 391, second pre-trained model 392, third pre-trained model 393, fourth pre-trained model 394, and fifth pre-trained model 396 described in the first embodiment are not limited to deep learning models, but may be other machine learning models (for example, supervised learning models such as SVM (Support Vector Machine)).

Furthermore, the control device 30 described in the first embodiment does not necessarily have to include a coordinate transformation unit 33 and a correction unit 34. In this case, the acquisition unit 31 only needs to send the acquired force data to the determination unit 35 and the control unit 36. Also in this case, when the acquisition unit 31 acquires the center position coordinates of the bolt 201, it only needs to send an acquisition flag to the counter unit 38 indicating that it has acquired the center position coordinates of the bolt 201 that is the target of the most recent construction. Then, the calculation unit 32 only needs to send the calculated position coordinates and attitude angle of the nut 202 to the determination unit 35 and the control unit 36. Then, the first determination unit 351 makes a first determination based on the force data received from the acquisition unit 31, the second determination unit 352 makes a second determination based on the position coordinates and attitude angle of the nut 202 received from the calculation unit 32, the third determination unit 353 makes a third determination based on the image received from the acquisition unit 31, the position coordinates and attitude angle of the nut 202 received from the calculation unit 32, and the force data received from the acquisition unit 31, the fourth determination unit 354 makes a fourth determination based on the image received from the acquisition unit 31, the position coordinates and attitude angle of the nut 202 received from the calculation unit 32, and the force data received from the acquisition unit 31, and the fifth determination unit 355 makes a fifth determination based on the force data received from the acquisition unit 31 if the result of the fourth determination received from the fourth determination unit 354 indicates that the fastening of the nut 202 was successful. Then, if the first determination result indicates that the nut 202 is in contact with the bolt 201, the first control unit 361 executes the fastening operation of the nut 202 to the bolt 201 based on a dataset including the position coordinates and orientation angle of the nut 202 received from the calculation unit 32 and the force data received from the acquisition unit 31. The second control unit 362 then controls the movement and rotation of the end effector 12 based on the image received from the acquisition unit 31 and a dataset including the position coordinates and orientation angle of the nut 202 received from the calculation unit 32.

Furthermore, the control device 30 described in the second embodiment may further include a current value conversion unit (not shown) that receives current values from the acquisition unit 31 and converts the received current values into force data including contact force and torque. In this case, the current value conversion unit can, for example, send the force data including contact force and torque converted from the current values to a correction unit 34 or the like. The current value conversion unit may, for example, convert the current values into force data based on correspondence relationship information (such as a function) that associates the current values of the currents flowing through each motor 117 (first motor 117a to sixth motor 117f) with the contact force and torque. In this case, the correspondence relationship information is, for example, stored in advance in a storage unit 39, and the current value conversion unit can perform the above conversion by referring to the correspondence relationship information stored in the storage unit 39 in a timely manner.

Furthermore, the first determination unit 351 may also use the image received from the acquisition unit 31 when performing the first determination. That is, the first determination unit 351 may perform the first determination based on the image and force data. In this case, the first determination unit 351 performs the determination using a predetermined trained model that has been trained to output the result of the first determination when a dataset containing the image and force data is input. That is, the trained model takes the dataset containing the image and force data as input elements and the result of the first determination as its output element. This trained model may, for example, be pre-stored in the storage unit 39, and the first determination unit 351 may obtain the output result of the first determination by inputting the dataset into the trained model stored in the storage unit 39.

Furthermore, the second determination unit 352 may also use the image received from the acquisition unit 31 when performing the second determination. That is, the second determination unit 352 may perform the second determination based on the image and the position coordinates and orientation angle of the nut 202. In this case, the second determination unit 352 performs the determination using a predetermined trained model that has been trained to output the result of the second determination when a dataset including the image and the position coordinates and orientation angle of the nut 202 is input. That is, the trained model takes the dataset including the image and the position coordinates and orientation angle of the nut 202 as input elements and the result of the second determination as its output element. The trained model may, for example, be pre-stored in the storage unit 39, and the second determination unit 352 may obtain the output result of the second determination by inputting the dataset into the trained model stored in the storage unit 39.

Furthermore, the third determination unit 353 does not need to use the image received from the acquisition unit 31 when determining whether the nut 202 is successfully threaded. In this case, the third trained model 393 takes a dataset containing the position coordinates and orientation angle of the nut 202 and force data as input elements, and outputs a determination result indicating whether the nut 202 and bolt 201 are successfully threaded.

Furthermore, the fifth determination unit 355 may, based on the force data received from the correction unit 34, simultaneously perform a determination regarding cutting—whether the torque required for cutting has exceeded a third threshold—and a determination regarding the drive of the end effector 12—whether the fastening motor 120a has changed from a stopped state to a rotating state—before performing the fifth determination. In other words, the fifth determination unit 355 may perform the fifth determination if the cutting determination indicates that the data (contact force) has exceeded the third threshold, and the determination regarding the drive of the end effector 12 indicates that the fastening motor 120a is in a rotating state. The third threshold is pre-stored, for example, in the storage unit 39, and the fifth determination unit 355 performs the cutting determination by referencing the third threshold stored in the storage unit 39 in a timely manner.

(Computer configuration)
Figure 17 is a hardware configuration diagram showing the configuration of the computer 1100 according to this embodiment. The computer 1100 includes, for example, a processor 1110, a main memory 1120, a storage 1130, and an interface 1140.

The control device 30 described above is implemented in, for example, one or more computers 1100. The operation of each processing unit described above is stored in the storage 1130 in the form of a program. The processor 1110 reads the program from the storage 1130, loads it into the main memory 1120, and executes the above processing according to the program. The processor 1110 also reserves a storage area in the main memory 1120 corresponding to the storage unit 39 described above, according to the program. The program may be for realizing a part of the functions to be performed by the computer 1100. For example, the program may perform functions in combination with other programs already stored in the storage 1130, or in combination with other programs implemented in other devices. In addition to the above configuration, or in place of the above configuration, the computer 1100 may be equipped with a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device). Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor 1110 may be realized by the integrated circuit. Examples of storage 1130 include magnetic disks, magneto-optical disks, and semiconductor memory. Storage 1130 may be an internal medium directly connected to the bus of the computer 1100, or an external medium connected to the computer 1100 via an interface 1140 or a communication line. Furthermore, if this program is distributed to the computer 1100 via a communication line, the computer 1100 that receives the distribution may expand the program into the main memory 1120 and execute the above processing. In the above embodiment, storage 1130 is a tangible storage medium that is not temporary. Furthermore, the program may be for realizing some of the functions described above. Furthermore, the program in question may be a so-called differential file (differential program) that implements the aforementioned functions in combination with other programs already stored in the storage 1130.

<Note>
The assembly system, control method, and program described in each embodiment can be understood, for example, as follows:

(1) The assembly system 100 according to the first embodiment comprises a manipulator 10 having an arm 11 having a plurality of joints 110, an end effector 12 that is moved by the arm 11 and holds a nut 202 and rotates the nut 202 around an axis (rotation axis Ar or axis Ar1), a first sensor 13 (encoder 130) that acquires the joint angle of each of the joints 110, and a second sensor 14 (force sensor 140, current sensor 141) that acquires the contact force and torque transmitted from the end effector 12 when the nut 202 comes into contact with a bolt 201 provided on the object to be constructed 200, and a control device 30 that controls the manipulator 10, the control device 30 controls each of the aforementioned joints The system comprises: an acquisition unit 31 that acquires the joint angle of the joint portion 110, the contact force, and the torque at a first predetermined period; a calculation unit 32 that calculates the position coordinates and attitude angle of the nut 202 based on the acquired joint angles of each of the joint portions 110; and a control unit 36 that controls the end effector 12 using a first trained model 391 that, upon input of the position coordinates and attitude angle of the nut 202 at a first time point and the contact force and torque at the first time point, outputs command values indicating the position coordinates and attitude angle of the nut 202 at a second time point, which is later than the first time point and approaches the target position coordinates and attitude angle of the nut 202 indicating that the nut 202 is fastened to the bolt 201.

This allows the state of the nut 202 to transition toward the state where the nut 202 is fastened to the bolt 201 each time an input is received to the first trained model 391.

(2) The assembly system 100 according to the second embodiment is the assembly system 100 of (1), wherein the first trained model 391 may be trained to output the command value corresponding to the input by repeatedly performing a training step in which a training dataset is input, which includes a plurality of position coordinates and orientation angles of the nuts 202 acquired during a teaching operation that controls the end effector 12 so that the known position coordinates and orientation angles of the nuts 202, which are randomly deviated by a given amount from the target position coordinates and target orientation angles of the nuts 202, become the target position coordinates and target orientation angles, and the contact force and torque corresponding to the time when each position coordinate and orientation angle of the nuts 202 was acquired.

This allows the above effects to be realized with more specific settings. Furthermore, for example, even if a deviation in the positional coordinates or orientation angle of the nut 202 relative to the bolt 201 occurs during actual operation, it is possible to suppress the deviation of the actual fastening time from the predicted fastening time.

(3) The assembly system 100 according to the third embodiment is the assembly system 100 of (1), further comprising an imaging device 20 provided on the end effector 12 and generating an image of the object to be constructed 200, wherein the acquisition unit 31 further acquires the image at a second predetermined period, and the control unit 36 controls the end effector 12 using a second trained model 392 that has been learned to output command values indicating the position coordinates and orientation angle of the nut 202 at a fourth time point, which is later than the third time point, when the nut 202 approaches the second position where it contacts the bolt 201, upon input of the image at a third time point acquired at a first position where the nut 202 is not in contact with the bolt 201, and the position coordinates and orientation angle of the nut 202 at the third time point.

This allows the nut 202, currently located at the first position, to move towards the second position each time an input is received for the second trained model 392.

(4) The assembly system 100 according to the fourth embodiment is the assembly system 100 of (3), wherein the second trained model 392 may be trained to output the command value corresponding to the input by repeatedly performing a training step in which a training dataset is input, which includes a plurality of images acquired during a direct teaching operation to move the end effector 12 from the second position to the first position, and the position coordinates and attitude angle of the nut 202 corresponding to the time when each of the images was acquired.

This allows for the acquisition of higher-quality training datasets with a lower proportion of noise and other unwanted elements.

(5) The assembly system 100 according to the fifth embodiment is the assembly system 100 of (3) or (4), wherein the control unit 36 controls the end effector 12 using a third trained model 393 that has been trained to output a determination result indicating the success or failure of screwing the nut 202 and the bolt 201 when the image acquired at the second position, the position coordinates and orientation angle of the nut 202 corresponding to the time the image was acquired, and the contact force and torque corresponding to the time the image was acquired are input.

This prevents, for example, incorrect fastening of the nut 202 and bolt 201.

(6) The assembly system 100 according to the sixth embodiment is any one of the assembly systems 100 from (1) to (4), wherein the position coordinates of the nut 202 are expressed in a world coordinate system based on the origin of the manipulator 10, the control device 30 further comprises a coordinate transformation unit 33 that transforms the calculated position coordinates of the nut 202 from the world coordinate system to the bolt center coordinate system, and the control unit 36 may input the transformed position coordinates of the nut 202 into the first learned model 391.

This allows us to represent the relative position coordinates of the nut 202 with respect to the bolt 201 using the same scale, such as the bolt 201's center coordinates, whereas the position coordinates of the nut 202, expressed in the world coordinate system, are unique position coordinates that occur for each bolt 201.

(7) The assembly system 100 according to the seventh embodiment is one of the assembly systems 100 from (1) to (4), wherein the control device 30 further comprises a correction unit 34 that corrects the acquired contact force using a correction value relating to the self-weight of the end effector 12 based on the joint angle of each of the acquired joints 110, and the control unit 36 may input the corrected contact force to the first learned model 391.

This allows the input element of the first trained model 391 to be a contact force that is closer to the actual contact force than the contact force transmitted from the end effector 12.

(8) The assembly system 100 according to the eighth embodiment is any one of the assembly systems 100 from (1) to (4), and the second sensor 14 may be a current sensor 141 capable of acquiring the current value of the motor 117 of each of the joint portions 110.

This allows for a reduction in the overall size and weight of the manipulator 10 compared to, for example, using a force sensor 140 as the second sensor 14.

(9) The assembly system 100 according to the ninth embodiment is one of the assembly systems 100 from (1) to (4), further comprising a base portion 15 on which the manipulator 10 is mounted, and a moving mechanism 16 (wheels 160, electric slider 161) that allows the base portion 15 to move around the object to be constructed 200.

This allows the manipulator 10 to operate over a wider range.

(10) An assembly system 100 according to the tenth embodiment is the assembly system 100 of (3) or (4), wherein the end effector 12 has a base portion 122 and a nut runner 121 having a base portion 122 and a socket portion 123 that extends from the base portion 122 and is cylindrical in shape around the axis, with a housing portion 123r formed on the tip side that can accommodate at least a part of the nut 202 and a part that is rotatably fixed to the outer circumference of the socket portion 123 and extends in a direction intersecting the axis The nut holding member 124 may have a first portion 126, a second portion 127 having an extended portion 127a extending from the first portion 126 toward the tip, and a claw portion 127b provided at the tip of the extended portion 127a that protrudes from the extended portion 127a toward the axis and contacts the nut 202; and an elastic member 128 provided between the base portion 122 and the first portion 126, which imparts an elastic force to the first portion 126 toward the tip.

This allows the end effector 12 to hold the nut 202 with a simpler configuration compared to, for example, an end effector 12 equipped with a mechanism for suctioning and holding the nut 202.

(11) The assembly system 100 according to the eleventh embodiment is the assembly system 100 of (10), wherein the imaging device 20 is located in the socket portion 123 on the side closer to the base portion 122 than the housing portion 123r, and the imaging device 20 may include an imaging unit 21 that generates the image of the object to be constructed 200 by imaging it through the hole (nut hole 202h) of the nut 202, and an illumination unit 22 that irradiates light toward the object to be constructed 200 through the hole (nut hole 202h) of the nut 202.

This allows for the generation of images while concentrating illumination on the imaging range of the imaging device 20.

(12) A control method relating to the twelfth aspect is a control method for controlling a manipulator 10 having an arm 11 having a plurality of joints 110, an end effector 12 which is moved by the arm 11 and holds a nut 202 and rotates the nut 202 around its axis, a first sensor 13 which acquires the joint angle of each of the joints 110, and a second sensor 14 which acquires the contact force and torque transmitted from the end effector 12 when the nut 202 comes into contact with a bolt 201 provided on the object to be constructed 200, wherein the control method acquires the joint angle of each of the joints 110, the contact force and torque at a first predetermined period. The process involves: 1) calculating the position coordinates and orientation angle of the nut 202 based on the acquired joint angles of each of the joint portions 110; and 2) controlling the end effector 12 using a first trained model 391, which is trained to output command values indicating the position coordinates and orientation angle of the nut 202 at a second time point, after the first time point, when the position coordinates and orientation angle of the nut 202 at a first time point and the contact force and torque at the first time point are input.

(13) The program according to the 13th embodiment provides the computer 1100 of a control device 30 that controls a manipulator 10 having an arm 11 having a plurality of joints 110, an end effector 12 that is moved by the arm 11 and holds a nut 202 and rotates the nut 202 around its axis, a first sensor 13 that acquires the joint angle of each of the joints 110, and a second sensor 14 that acquires the contact force and torque transmitted from the end effector 12 when the nut 202 comes into contact with a bolt 201 provided on the object to be constructed 200, to the computer 1100 of the control device 30, which provides the joint angle of each of the joints 110, the contact force and torque in a first predetermined period The following steps are performed: acquiring information; calculating the position coordinates and orientation angle of the nut 202 based on the acquired joint angles of each of the joint portions 110; and controlling the end effector 12 using a first trained model 391, which is trained to output command values indicating the position coordinates and orientation angle of the nut 202 at a second time point (later than the first time point) when the position coordinates and orientation angle of the nut 202 at a first time point, and the contact force and torque at the first time point, are input.

10...Manipulator 11...Arm 12...End effector 13...First sensor 14...Second sensor 15...Base 16...Movement mechanism 17...Calibration mechanism 20...Imaging device 21...Imaging unit 22...Illumination unit 30...Control device 31...Acquisition unit 32...Calculation unit 33...Coordinate transformation unit 34...Correction unit 35...Determination unit 36...Control unit 37...Learning unit 38...Counter unit 39...Storage unit 100...Assembly system 110...Joint 111...First joint 112...Second joint 113...Third joint 114...Fourth joint 115...Fifth joint 116...Sixth joint 117...Motor 117a...First motor 117b...Second motor 117c...Third motor 117d...Fourth motor 117e...5th motor 117f...6th motor 118...connection part 118a...1st connection part 118b...2nd connection part 118c...3rd connection part 120...main body part 120a...fastening motor 121...nut runner 122...base part 122a...front surface 123...socket part 123a...tapered surface 123b...fastening surface 123c...positioning surface 123e...opening 123i...inner circumferential surface 123o...outer circumferential surface 123r...housing part 124...nut holding member 125...rotating shaft 126...1st part 127...2nd part 127a...extension part 127b...claw part 128...elastic member 130...encoder 131...1st encoder 132...2nd encoder 133...3rd encoder 134...Fourth encoder 135...Fifth encoder 136...Sixth encoder 140...Force sensor 141...Current sensor 160...Wheel 161...Electric slider 200...Object to be constructed 201...Bolt 201a...Male threaded part 201e, 202e...End face 202...Nut 202a...Nut body 202b...Collar 202h...Nut hole 210...First part 220...Second part 300...Workbench 351...First judgment unit 352...Second judgment unit 353...Third judgment unit 354...Fourth judgment unit 355...Fifth judgment unit 356...Sixth judgment unit 361...First control unit 362...Second control unit 363...Third control unit 391...First trained model 392...Second trained model 393…Third trained model 394…Fourth trained model 395…CAD data 396…Fifth trained model 397…Variation generation model 1100…Computer 1110…Processor 1120…Main memory 1130…Storage 1140…Interface Ar…Rotation axis Ar1…Axis Ar2…Centerline CG…Center of gravity of end effector Da…Axis direction Da1…One side Da2…Other side Dv…Vertical direction Fb…Contact force Fg…Self-weight Fg1…Correction value Ft…Torque O1…First rotation axis O2…Second rotation axis O3…Third rotation axis O4…Fourth rotation axis O5…Fifth rotation axis O6…Sixth rotation axis R…Accommodation space S…Controlled device

Claims (13)

An arm having multiple joints,
An end effector which is moved by the aforementioned arm and holds the nut and rotates the nut around its axis,
A first sensor for acquiring the joint angle of each of the aforementioned joints,
A second sensor acquires the contact force and torque transmitted from the end effector when the nut comes into contact with a bolt provided on the object to be constructed,
A manipulator having,
A control device for controlling the manipulator,
Equipped with,
The control device is
An acquisition unit that acquires the joint angle of each of the aforementioned joints, the contact force and torque at a first predetermined period,
A calculation unit calculates the position coordinates and orientation angle of the nut based on the joint angles of each of the aforementioned joints obtained,
A control unit controls the end effector using a first trained model that, upon input of the position coordinates and orientation angle of the nut at a first time point, and the contact force and torque at the first time point, outputs command values indicating the position coordinates and orientation angle of the nut at a second time point, which is later than the first time point and approaches the target position coordinates and orientation angle of the nut indicating that the nut is fastened to the bolt.
An assembly system equipped with the following features. The first pre-trained model described above is:
A plurality of position coordinates and orientation angles of the nuts obtained during a teaching operation that controls the end effector so that the known position coordinates and orientation angles of the nuts, which are randomly deviated by a given amount from the target position coordinates and target orientation angles of the nuts, are the target position coordinates and target orientation angles,
The contact force and torque corresponding to the time at which the position coordinates and orientation angle of each nut were acquired,
The assembly system according to claim 1, wherein a learning step in which a training dataset including is input is repeatedly performed is learned to output the command value corresponding to the input. The end effector is further provided with an imaging device that generates an image of the object to be constructed,
The acquisition unit further acquires the image at a second predetermined cycle,
The control unit,
The assembly system according to claim 1, wherein when the image at a third time taken at a first position in which the nut is not in contact with the bolt, and the position coordinates and attitude angle of the nut at the third time are input, the end effector is controlled using a second trained model that has been trained to output command values indicating the position coordinates and attitude angle of the nut at a fourth time, which is later than the third time, when the nut has approached a second position in which it is in contact with the bolt. The second pre-trained model described above is:
A plurality of images acquired during the direct teaching operation that moves the end effector from the second position to the first position,
The position coordinates and orientation angle of the nut corresponding to the time when each of the aforementioned images was acquired,
The assembly system according to claim 3, wherein a learning step in which a training dataset including is input is repeatedly performed is learned to output the command value corresponding to the input. The control unit,
The assembly system according to claim 3 or 4, wherein the end effector is controlled using a third trained model that has been trained to output a determination result indicating whether the nut and bolt are screwed together, when the image acquired at the second position, the position coordinates and orientation angle of the nut corresponding to the time the image was acquired, and the contact force and torque corresponding to the time the image was acquired are input. The position coordinates of the nut are expressed in a world coordinate system that is based on the origin of the manipulator.
The control device further includes a coordinate transformation unit that transforms the calculated position coordinates of the nut from the world coordinate system to the bolt center coordinate system.
The assembly system according to any one of claims 1 to 4, wherein the control unit inputs the converted position coordinates of the nut to the first trained model. The control device further includes a correction unit that corrects the acquired contact force using a correction value relating to the self-weight of the end effector based on the joint angle of each of the acquired joints,
The assembly system according to any one of claims 1 to 4, wherein the control unit inputs the corrected contact force to the first learned model. The assembly system according to any one of claims 1 to 4, wherein the second sensor is a current sensor capable of acquiring the current value of the motors of each of the joints. The base on which the manipulator is placed,
A movable mechanism that allows the base portion to move around the object to be constructed,
The assembly system according to any one of claims 1 to 4, further comprising: The aforementioned end effector is,
A nut runner having a base and a socket portion that extends from the base and is cylindrical in shape centered on the axis, with a receiving portion formed at the tip side capable of accommodating at least a part of the nut,
A nut holding member having a first portion that is rotatably fixed to the outer circumference of the socket portion and extends in a direction intersecting the axis, and a second portion that has an extended portion extending from the first portion toward the tip and a claw portion provided at the tip of the extended portion that protrudes from the extended portion so as to approach the axis and contacts the nut,
An elastic member provided between the base and the first portion, which imparts an elastic force to the first portion toward the tip side,
The assembly system according to claim 3 or claim 4, having the following features. The imaging device is positioned in the socket portion on the side closer to the base portion than the housing portion.
The imaging device is
An imaging unit generates the image of the object to be constructed by taking an image of it through the hole in the nut,
A lighting unit that shines light towards the object to be constructed through the hole in the nut,
The assembly system according to claim 10, having the following features. An arm having multiple joints,
An end effector which is moved by the aforementioned arm and holds the nut and rotates the nut around its axis,
A first sensor for acquiring the joint angle of each of the aforementioned joints,
A second sensor acquires the contact force and torque transmitted from the end effector when the nut comes into contact with a bolt provided on the object to be constructed,
A control method for controlling a manipulator having,
A step of acquiring the joint angle of each of the aforementioned joints, the contact force and torque at a first predetermined period,
A step of calculating the position coordinates and orientation angle of the nut based on the joint angles of each of the joints obtained above,
Steps include: controlling the end effector using a first trained model that, upon inputting the position coordinates and orientation angle of the nut at a first time point, and the contact force and torque at the first time point, outputs command values indicating the position coordinates and orientation angle of the nut at a second time point, which is later than the first time point and approaches the target position coordinates and orientation angle of the nut indicating that the nut is fastened to the bolt;
A control method for executing this. An arm having multiple joints,
An end effector which is moved by the aforementioned arm and holds the nut and rotates the nut around its axis,
A first sensor for acquiring the joint angle of each of the aforementioned joints,
A second sensor acquires the contact force and torque transmitted from the end effector when the nut comes into contact with a bolt provided on the object to be constructed,
In the computer of a control device that controls a manipulator having,
A step of acquiring the joint angle of each of the aforementioned joints, the contact force and torque at a first predetermined period,
A step of calculating the position coordinates and orientation angle of the nut based on the joint angles of each of the joints obtained,
Steps include: controlling the end effector using a first trained model that, upon inputting the position coordinates and orientation angle of the nut at a first time point, and the contact force and torque at the first time point, outputs command values indicating the position coordinates and orientation angle of the nut at a second time point, which is later than the first time point and approaches the target position coordinates and orientation angle of the nut indicating that the nut is fastened to the bolt;
A program that executes the command.