JP7591938B2 - Control device - Google Patents

Description

The present invention relates to a control device for controlling a robot.

Conventionally, a so-called hand-eye type robot system has been known that combines a robot with a three-dimensional camera fixed to the robot hand (see, for example, Patent Document 1). This type of robot system requires calibration before operation to convert the camera coordinates into the robot coordinates of the robot hand. In Patent Document 1, one calibration mark is placed in the robot coordinate system, and the calibration mark is observed at multiple observation points where the three-dimensional camera is translated while facing the calibration mark, and at multiple observation points where the calibration mark is viewed from a different direction from these observation points, and calibration data is obtained by associating the mark position recognition data obtained at each observation point with the posture data of the robot hand.

JP 2012-91280 A

In Patent Document 1, it is necessary to prepare a calibration mark when performing calibration. The calibration mark is different in material, shape, size, etc. from the workpiece that is picked up when the robot system is in operation, so it is necessary to image the calibration mark by changing the camera's imaging parameters (exposure time, gain, etc.) and focus position from when the workpiece is imaged. In other words, since calibration is performed with parameters that are different from those used during operation, it is difficult to say that the calibration is optimal for operation.

One solution is to use the workpiece as the calibration mark instead of the calibration mark. However, when determining the parameters for converting the camera coordinates into the robot coordinates, it is necessary to rotate the robot hand. In particular, rotating the robot hand more widely increases accuracy in principle, so there is a demand to rotate the robot hand more widely during calibration.

However, the camera's field of view and the workpiece are small compared to the range of motion of the robot hand, so if the robot hand is rotated significantly, the workpiece will go outside the camera's field of view, making it impossible to obtain calibration data.

Even if the workpiece does not leave the camera's field of view when the robot hand is rotated, if only a portion of the workpiece is imaged, sufficient data for accurate calibration cannot be obtained.

Furthermore, generally speaking, measurement accuracy is lower in the peripheral areas of a 3D camera's field of view than in the area near the center. If the robot hand is rotated significantly, calibration will be performed using data from the peripheral areas of the 3D camera's field of view, and in this respect too, there is a risk that accurate calibration will not be possible.

The present disclosure has been made in consideration of these points, and its purpose is to make it possible to obtain accurate calibration data when performing calibration using a workpiece.

To achieve the above object, a first aspect of the present disclosure can be based on a control device that measures the shape of a workpiece using a three-dimensional camera fixed to a robot hand, and performs calibration to convert the camera coordinates of the three-dimensional camera into the robot coordinates of the robot hand based on the measured shape data of the workpiece. The control device includes a calibration execution unit that can perform normal calibration processing and high-precision calibration processing.

In the normal calibration process, the robot hand is moved in the XYZ directions and rotated multiple times around a predetermined axis. After each operation, the workpiece is measured by the three-dimensional camera at the stop position of the robot hand, and workpiece coordinates are detected based on the shape data obtained. The robot coordinates at the time the workpiece coordinates are obtained are also obtained. Based on the multiple pairs of workpiece coordinates and robot coordinates obtained, parameters for converting the position in the camera coordinates to a position in the robot coordinates can be calculated.

On the other hand, in the high-precision calibration process, a provisional rotation center is determined based on the relative position between the tool center of the robot hand and the three-dimensional camera obtained by the normal calibration process, and the robot hand is rotated multiple times around the provisional rotation center. At each position, the workpiece is measured by the three-dimensional camera to detect workpiece coordinates based on shape data obtained, and the robot coordinates at the time the workpiece coordinates are obtained are obtained. The final rotation center of the robot hand can be determined based on the multiple pairs of workpiece coordinates and robot coordinates obtained.

This configuration can be based on a robot system in which a three-dimensional camera that measures the three-dimensional shape of the workpiece is fixed to the robot hand. Therefore, when the robot system is in operation, the position of the workpiece can be measured by the three-dimensional camera without having to significantly retract the robot hand, and the three-dimensional shape of the workpiece can be measured from close to the workpiece to be measured, improving measurement accuracy. The subject of the present invention may be a robot system or a robot control device (robot controller).

When the three-dimensional camera is fixed to the robot hand, it is necessary to perform calibration to convert the camera coordinates of the three-dimensional camera into the robot coordinates of the robot hand. In this configuration, a multi-stage calibration process is performed using the work to be picked during operation.

That is, first, as a normal calibration process, the robot hand is repeatedly moved in the XYZ directions and rotated, and the workpiece after each operation is measured with a three-dimensional camera to detect the workpiece coordinates, and the robot coordinates at the time the workpiece coordinates are obtained are also obtained. Based on the multiple pairs of workpiece coordinates and robot coordinates obtained in this way, parameters are calculated to convert the position in the camera coordinates to a position in the robot coordinates, and the relative positional relationship between the tool center of the robot hand and the three-dimensional camera is found.

After determining the relative positional relationship between the tool center of the robot hand and the three-dimensional camera, a high-precision calibration process is performed. First, a provisional rotation center of the robot hand is determined based on the relative position between the tool center of the robot hand and the three-dimensional camera determined as described above. This provisional rotation center is a rotation center that is temporarily set. By rotating the robot hand multiple times around the provisional rotation center, it is possible to prevent the workpiece from going outside the field of view of the three-dimensional camera, to ensure that the entire workpiece is in the field of view of the three-dimensional camera, and to place the workpiece near the center of the field of view of the three-dimensional camera. Then, at each position, the workpiece is measured by the three-dimensional camera to detect the workpiece coordinates, and the robot coordinates when the workpiece coordinates are obtained are obtained. Based on the multiple pairs of workpiece coordinates and robot coordinates obtained, the final rotation center of the robot hand can be determined with high precision.

The robot coordinates when the workpiece coordinates are obtained can be obtained, for example, based on the operating state of the robot hand, or from a robot controller that controls the robot.

In a second aspect of the present disclosure, the calibration execution unit rotates the robot hand around the provisional rotation center in the high-precision calibration process to a rotation angle that is greater than the rotation angle of the robot hand around a predetermined axis in the normal calibration process.

According to this configuration, by increasing the rotation angle of the robot hand in the high-precision calibration process, the accuracy of the calibration data obtained is increased. In the high-precision calibration process, the robot hand is rotated around the temporary rotation center, so even if the rotation angle is increased as described above, there is a low possibility that all or part of the workpiece will fall outside the field of view of the 3D camera.

In a third aspect of the present disclosure, the calibration execution unit executes a first process in the high-precision calibration process to determine the center of rotation of the robot hand based on a set of the multiple workpiece coordinates and the robot coordinates, and a second process in which, using the center of rotation determined in the first process as the next center of rotation, the robot hand is rotated multiple times, and at each position, workpiece coordinates are detected based on shape data obtained by measuring the workpiece with the three-dimensional camera, and the robot coordinates at the time the workpiece coordinates were obtained are obtained, and the final center of rotation of the robot hand is determined, based on the obtained sets of multiple workpiece coordinates and robot coordinates.

In other words, since the provisional center of rotation may deviate from the center that should ultimately be the center of rotation, instead of directly determining the final center of rotation based on this provisional center of rotation, a second process is executed to determine a new center of rotation for the robot hand using the center of rotation determined in the first process as the next center of rotation, and the robot hand is rotated using the newly determined center of rotation in the second process as the next center of rotation to determine the final center of rotation for the robot hand, thereby reducing the error in the final center of rotation.

In a fourth aspect of the present disclosure, a process of determining the rotation center of the robot hand using the rotation center determined in the second process as the next rotation center, and a process of determining the rotation center of the robot hand using the rotation center determined in the first process as the next rotation center are repeatedly performed, and then the final rotation center of the robot hand can be determined.

With this configuration, the process of finding the rotation center can be repeated multiple times before the final rotation center is determined, allowing the center to converge to a rotation center with less error.

In a fifth aspect of the present disclosure, the calibration execution unit can repeatedly execute the high-precision calibration process to obtain multiple rotation centers of the robot hand, and determine a final rotation center based on the multiple rotation centers obtained.

In other words, if the high-precision calibration process is completed only once, there is a risk that the final rotation center determined may not be accurate due to some factor, but by determining the final rotation center based on multiple rotation centers obtained by performing the high-precision calibration process multiple times as in this configuration, an accurate rotation center can be obtained. When determining the final rotation center, for example, an averaging process of the multiple rotation centers may be performed.

In a sixth aspect of the present disclosure, the calibration execution unit determines whether the multiple rotation centers obtained by repeatedly executing the high-precision calibration process satisfy a predetermined convergence condition, and when it is determined that the predetermined convergence condition is satisfied, the high-precision calibration process can be terminated.

In other words, if the multiple rotation centers obtained by repeatedly executing the high-precision calibration process do not converge, the obtained rotation center may not be accurate. In this configuration, the high-precision calibration process is executed until a predetermined convergence condition is satisfied, so an accurate rotation center can be obtained. An example of the predetermined convergence condition is that the change in error of the rotation center obtained by the high-precision calibration process is equal to or less than a certain amount.

In a seventh aspect of the present disclosure, the calibration execution unit can rotate the robot hand by a predetermined angle around a temporary rotation center during the high-precision calibration process, and can move the robot hand so that the workpiece after the predetermined angle rotation is within the field of view of the three-dimensional camera.

In other words, when the robot hand is rotated a specified angle around the temporary center of rotation, depending on the position and size of the workpiece, there is a risk that the workpiece will go out of the field of view of the 3D camera after the specified angle of rotation. However, with this configuration, the workpiece remains within the field of view of the 3D camera even after the specified angle of rotation, so accurate calibration data can be obtained.

In an eighth aspect of the present disclosure, the calibration execution unit can determine the XYZ coordinates of a representative point of the work based on three-dimensional shape data obtained by measuring the work with the three-dimensional camera, and the determined XYZ coordinates of the representative point can be set as work coordinates.

In a ninth aspect of the present disclosure, the calibration execution unit determines the XY coordinates of a representative point of the work based on two-dimensional shape data obtained by measuring the work using the three-dimensional camera, and determines the Z coordinate of the representative point from a plane equation, and can set the determined XYZ coordinates of the representative point as work coordinates.

In a tenth aspect of the present disclosure, the calibration execution unit detects work coordinates based on shape data obtained by measuring the work with the three-dimensional camera after moving the robot hand to a first position, and detects work coordinates based on shape data obtained by measuring the work with the three-dimensional camera after moving the robot hand to a second position having a different Z coordinate, and converts the work coordinates detected at the first position and the work coordinates detected at the second position into pixel positions independent of the angle of view of the three-dimensional camera.

In other words, because the Z-direction position of the three-dimensional camera differs when the robot hand is in the first position from when it is in the second position, the X-coordinate of the workpiece coordinates differs depending on the camera's angle of view, and accurate calibration data may not be obtained. With this configuration, even if the Z-direction position of the three-dimensional camera differs, the workpiece coordinates can be converted into pixel positions that are independent of the three-dimensional camera's angle of view, so accurate calibration data can be obtained.

In an eleventh aspect of the present disclosure, a calibration mode for performing calibration is configured to be selectable, and when the calibration execution unit detects that the calibration mode has been selected, the calibration execution unit sequentially executes the normal calibration process and the high-precision calibration process.

According to this configuration, for example, if an operational mode and a calibration mode are installed, when the calibration mode is selected before the operational mode, the normal calibration process is executed first, and then the high-precision calibration process is executed, eliminating the need for the user to switch from the normal calibration process to the high-precision calibration process.

In a twelfth aspect of the present disclosure, the calibration execution unit generates a user interface having a display area capable of updating and displaying the detection results of the work coordinates detected in the high-precision calibration process and the convergence value of the rotation center. The control device includes a display unit that displays the user interface generated by the calibration execution unit, and is therefore capable of presenting the change in the convergence value to the user.

As described above, a provisional rotation center is determined based on the relative position of the tool center of the robot hand and the three-dimensional camera obtained by normal calibration processing, and the final rotation center of the robot hand can be determined with high precision based on multiple pairs of work coordinates and robot coordinates obtained by rotating the robot hand multiple times around the provisional rotation center. This makes it possible to obtain accurate calibration data when performing calibration using a workpiece.

FIG. 1 is a schematic diagram showing a state in which a workpiece is transported using a robot system. FIG. 1 is a block diagram showing a configuration of a robot system. FIG. 1 is a perspective view showing an example of a three-dimensional camera. 1 is a schematic diagram showing an example of gripping a workpiece with a robot hand; 1A and 1B are schematic diagrams showing an example of gripping a workpiece having a cavity from the inside with a robot hand. 1 is a schematic diagram showing an example of a robot hand suctioning and gripping a plate-shaped workpiece; FIG. 13 is a flowchart showing a procedure for setting a search model. 11 is a flowchart showing a procedure for setting a hand model and a gripping position. FIG. 13 is a diagram showing an example of a flowchart of basic control of a robot. 11 is a flowchart showing an example of a calibration procedure. 10 is a flowchart showing a detailed procedure of a first step of calibration. FIG. 13 is a diagram illustrating an example of a user interface for calibration. FIG. 13 is a schematic diagram showing how the robot hand is translated in a first step. 13A to 13C are diagrams illustrating an example of a calibration user interface that is displayed when the robot hand is moved to a first position in the first step. FIG. 13 is a diagram showing an example of a calibration user interface that is displayed when the robot hand is moved to the 27th position in the first step. FIG. 11 is a side view showing the positional relationship between the workpiece and the three-dimensional camera during measurement in the first step. FIG. 13 is a diagram showing the relationship between the workpiece and the field of view of the three-dimensional camera during measurement in the first step. 11 is a table showing distances from the center of the camera to representative points used when correcting the angle of view. FIG. 13 is a diagram showing an example of a calibration user interface that is displayed upon completion of the first step. 10 is a flowchart showing a detailed procedure of a second step of calibration. FIG. 13 is a diagram showing an example of a calibration user interface displayed in a second step. 13A and 13B are diagrams illustrating the positional relationship between the field of view and the workpiece when the movement of the robot hand is enlarged in the second step. FIG. 13 is a diagram showing an example of a calibration user interface that is displayed upon completion of the second step. 13 is a flowchart showing a detailed procedure of a third step of calibration. FIG. 13 is a diagram showing an example of a calibration user interface displayed when high-precision calibration is set. 13A to 13C are diagrams illustrating the case where the robot hand is rotated around the provisional center of rotation in the third step. 13A to 13C are diagrams illustrating a case where the robot hand is rotated while keeping the workpiece in view in the third step. 11A and 11B are diagrams for explaining a case where a deviation between an original rotation center and a true rotation center position is obtained. 13A and 13B are diagrams illustrating a case where the original rotation center is updated. FIG. 13 is a diagram showing an example of a user interface for calibration that is displayed when high-precision calibration processing is completed.

The following describes in detail an embodiment of the present invention with reference to the drawings. Note that the following description of the preferred embodiment is essentially merely an example and is not intended to limit the present invention, its applications, or its uses.

Figure 1 is a schematic diagram showing an example of the configuration of a robot system 1000 equipped with a control device 100 according to an embodiment of the present invention. This Figure 1 shows an example of bulk picking in which a robot RBT grasps multiple workpieces WK arranged in a work space in a manufacturing plant for various products one by one, transports them to a stage STG installed in a predetermined location, and places them on the stage STG.

The robot system 1000 includes a robot RBT and a robot controller 6 that controls the robot RBT. The robot RBT is a general-purpose industrial robot with a base portion fixed to the floor of a factory or the like. The robot RBT is also called a manipulator, for example, and is configured to be capable of six-axis control. This robot RBT includes an arm portion ARM extending from the base portion, and a robot hand HND provided at the tip of the arm portion ARM. The arm portion ARM can be configured as a multi-joint type having multiple joint portions as movable portions. The robot hand HND can be moved to a desired position within the movable range by the operation of each joint portion of the arm portion ARM and the rotational operation of the arm portion ARM itself.

The robot hand HND is configured to be able to grasp the workpiece WK, and its structure is not particularly limited. For example, the robot hand HND may be a robot hand HND1 having a structure for pinching and grasping the outside of the workpiece WK as shown in FIG. 4A, a robot hand HND2 having a structure for holding the workpiece WK2 by inserting a claw into the inside of the workpiece WK2 having a hollow portion and spreading it as shown in FIG. 4B, or a robot hand HND3 for sucking and holding a plate-shaped workpiece WK3 as shown in FIG. 4C, and any of these robot hands may be used. In this specification, the term "hold" is used to include all examples, such as the method of pinching the outside of the workpiece WK shown in FIG. 4A, the method of inserting a claw into the hollow portion and spreading it, and the method of sucking.

The robot controller 6 controls the operation of the arm section ARM and the opening and closing operation (grasping operation) of the robot hand HND. The robot controller 6 also acquires information necessary for controlling the robot RBT from the control device 100 shown in FIG. 2. For example, the position and orientation of the workpiece WK, which is a number of parts randomly placed in the container BX shown in FIG. 1, is acquired using the three-dimensional camera 2, and the position and orientation of the workpiece WK are detected by the control device 100, and the robot controller 6 acquires this information.

A functional block diagram of the robot system 1000 is shown in FIG. 2. In addition to the robot RBT and the robot controller 6, the robot system 1000 shown in this figure includes a control device 100, an operation unit 4, and a robot operation tool 7. The robot system 1000 also includes a display unit 3. The robot controller 6 may include the control device 100, or the control device 100 may be configured independently and connected to the robot controller 6. The control device 100 may also include the operation unit 4 and the robot operation tool 7.

The operation unit 4 is a section for performing various settings. The control device 100 also includes a measurement unit 20, which captures an image of the workpiece WK and measures the position, orientation, etc. of the workpiece WK. Furthermore, the user can use the display unit 3 to perform various settings, check the operating status of the robot RBT, check the operating status of the control device 100, check the calibration described below, and the like. The display unit 3 can also be part of the control device 100.

On the other hand, the robot controller 6 is a well-known component configured to control the robot RBT according to signals output from the control device 100. The robot operation tool 7 sets the operation of the robot RBT. Note that, although the operation unit 4 and the robot operation tool 7 are separate components in the example of FIG. 2, they may be a common component.

The three-dimensional camera 2 is a component that captures images of the working space and the workpiece WK, and is called a robot vision or the like. The three-dimensional camera 2 is fixed to the robot hand HND, so that the relative positional relationship between the three-dimensional camera 2 and the robot hand HND does not change. Therefore, the robot system 1000 of this embodiment is a robot system of a type called a hand-eye.

The objects to be imaged by the three-dimensional camera 2 include at least the workpiece WK, and may also include the container BX. Depending on the workpiece WK, the direction of the optical axis of the three-dimensional camera 2 can be set so that the three-dimensional camera 2 can image the area below when the robot hand HND is in a position and posture to pick up the workpiece WK below.

It is possible to obtain three-dimensional shape data (image data) showing the three-dimensional shape of the workpiece WK from the images captured by the three-dimensional camera 2. Three-dimensional shape data showing the three-dimensional shape of the container BX can also be obtained from the images captured by the three-dimensional camera 2. It is also possible to obtain two-dimensional shape data showing the two-dimensional shape of the workpiece WK from the images captured by the three-dimensional camera 2. The three-dimensional camera 2 can also output color images or black and white images of the workpiece WK.

Methods for acquiring the three-dimensional shape of the workpiece WK or container BX include, for example, the pattern projection method, stereo method, lens focusing method, light section method, optical radar method, interferometry (white light interferometry), and TOF method, and any of these methods may be used. Each method is well known in the art, so detailed explanations are omitted. In this embodiment, of the pattern projection methods, the phase shift method is used, in which pattern light having a periodic illuminance distribution is irradiated onto the object to be imaged and the light reflected from the surface of the object to be imaged is received.

The measurement unit 20 is made up of the three-dimensional camera 2 and the shape measurement processing unit 200. The shape measurement processing unit 200 can also be included as part of the three-dimensional camera 2. The shape measurement processing unit 200 can measure not only the position, posture, shape and dimensions of each part of the workpiece WK based on image data acquired by the three-dimensional camera 2, but also the position, shape and dimensions of the container BX. The shape measurement processing unit 200 can be part of the control device 100, but can also be configured as a separate component.

The measuring unit 20 is a three-dimensional measuring unit capable of performing three-dimensional measurements including the Z direction (height direction) in addition to the X and Y directions. The specific configuration of the three-dimensional camera 2 equipped in the measuring unit 20 is determined according to the measurement technology of the three-dimensional shape. In this example, the three-dimensional camera 2 is equipped with a camera, lighting, projector, etc. For example, when measuring the three-dimensional shape of the workpiece WK by the phase shift method, the three-dimensional camera 2 is equipped with a projector (light-projecting unit) PRJ and multiple cameras (light-receiving units) CME1, CME2, CME3, and CME4 as shown in FIG. 3. The projector PRJ is a member that irradiates light onto the workpiece WK. The cameras CME1, CME2, CME3, and CME4 are members having image sensors that receive light projected by the projector PRJ and reflected on the surface of the workpiece WK. The light source of the light-projecting unit can be composed of, for example, multiple light-emitting diodes, a liquid crystal panel, an organic EL panel, a digital micromirror device (DMD), etc. The number of cameras and lights that the three-dimensional camera 2 has is not particularly limited. Furthermore, the connections between the three-dimensional camera 2, the robot RBT, and the robot controller 6 are not limited to wired connections, and may be well-known wireless connections.

The three-dimensional camera 2 may be composed of multiple components such as the cameras CME1, CME2, CME3, and CME4 and the projector PRJ, or may be composed of these as an integrated unit. For example, the three-dimensional camera 2 may be a 3D imaging head formed by integrating the cameras CME1, CME2, CME3, and CME4 and the projector PRJ into a head shape.

The three-dimensional shape data itself can also be generated by the three-dimensional camera 2. In this case, the three-dimensional camera 2 is provided with an image processing IC or the like that realizes the function of generating three-dimensional shape data. Alternatively, the three-dimensional shape data may not be generated on the shape measurement processing unit 200 side, but rather the raw images captured by the three-dimensional camera 2 may be image-processed in another part of the control device 100 to generate three-dimensional shape data such as a three-dimensional image.

The control device 100 performs three-dimensional searches, interference determination, gripping solution calculations, etc. based on three-dimensional shape data such as the position and orientation of the workpiece WK obtained by the measurement unit 20. This control device 100 can be configured with a general-purpose computer with a dedicated image processing program installed, a specially designed image processing controller, and dedicated hardware. It can also be configured with an image processing program installed on a dedicated computer with hardware such as a graphics board specialized for image inspection processing.

In the example of FIG. 2, the three-dimensional camera 2 and the shape measurement processing unit 200 provided in the measurement unit 20 are shown as separate components, but the present invention is not limited to this configuration. For example, the three-dimensional camera 2 and the shape measurement processing unit 200 can be integrated, or part of the three-dimensional camera 2 can be incorporated into the shape measurement processing unit 200, or part of the shape measurement processing unit 200 can be incorporated into the three-dimensional camera 2. In this way, the division of the components shown in FIG. 2 is only one example, and multiple components can be integrated or one component can be separated. For example, the operation unit 4 and the robot operation tool 7 can be a common component.

The display unit 3 can be, for example, an LCD monitor, an organic EL display, or a CRT. The operation unit 4 is a component for performing various settings such as various simulations and image processing, and can use input devices such as a keyboard and a mouse. The display unit 3 can also be integrated with the operation unit 4 by using a touch panel.

For example, if the control device 100 is configured as a computer with an image processing program installed, a graphical user interface (GUI) screen generated by executing the image processing program is displayed on the display unit 3, as described below. Various settings can be made from the GUI displayed on the display unit 3, and processing results, etc. can be displayed. In this case, the display unit 3 can also be used as a setting unit or input unit for making various settings, and the setting unit and input unit may include the operation unit 4.

The robot controller 6 controls the robot's operation based on, for example, information captured by the three-dimensional camera 2 and information output from the control device 100. The robot operation tool 7 is a member for setting the operation of the robot RBT, and a pendant or the like can be used.

Based on the three-dimensional shape of the workpiece WK obtained by the three-dimensional camera 2, the robot controller 6 controls the robot RBT to identify the workpiece WK to be grasped from among the multiple workpieces WK and grasp the workpiece WK. Then, while grasping the workpiece WK at the grasping position, the arm unit ARM is operated to move the workpiece WK to a predetermined placement position, for example, on the stage STG, and place the workpiece WK in a predetermined posture. In other words, the robot controller 6 controls the operation of the robot RBT to grasp the workpiece WK to be picked (grasped) identified by the three-dimensional camera 2 and the control device 100 with the robot hand HND, place the grasped workpiece WK in a predetermined reference posture on a placement location (stage STG), and release the robot hand HND. The stage STG can be, for example, a conveyor belt or a pallet. The grasping operation can also be called picking, and the grasping position can also be called the picking position. Additionally, the act of placing can also be called "place," and the placement position can also be called "place position."

Here, the picking targeted by this embodiment is not limited to bulk picking, and the robot RBT can transport the workpieces WK regardless of how they are placed. In this specification, bulk picking is used to mean not only the case where the workpieces WK that are randomly stacked in a container BX are grasped by the robot RBT and placed in a predetermined position, but also the case where the workpieces WK that are stacked in a predetermined area without using a container are grasped and placed, or the case where the workpieces WK that are arranged and stacked in a predetermined posture are grasped and placed one by one. In addition, the workpieces WK do not necessarily have to be stacked on top of each other, and the workpieces WK that are randomly placed on a plane without overlapping each other are also called bulk picking in this specification (this is the same reason why the workpieces WK are still called bulk picking even when they are picked one by one and there is no overlapping at the end of the picking). Note that the present invention is not necessarily limited to bulk picking, and can also be applied to the application of picking up workpieces WK that are not stacked.

When performing bulk picking operations with the robot system 1000, teaching can be performed in advance, including settings for performing the bulk picking operations. Specifically, it is possible to register which part of the workpiece WK the robot hand HND will grasp and in what posture, the grasping position and posture when grasping, and the placement position and posture when placing can be registered. Such settings can be performed with a robot operating tool 7 such as a pendant. It is also possible to perform settings in the vision space without operating the actual robot RBT.

The display unit 3 displays a work model that virtually represents the three-dimensional shape of the workpiece WK, and a robot hand model constructed from three-dimensional CAD data that virtually represents the three-dimensional shape of the robot hand HND, in three-dimensional form in a virtual three-dimensional space. Furthermore, the display unit 3 can also display a basic orientation image of the work model as a six-sided view. This makes it possible to display each posture of the work model in a six-sided view and perform the task of setting the gripping position.

(Settings Procedure)
5 is a flowchart showing the procedure for setting the search model. In step SB1, a search model of the workpiece WK is registered. The search model of the workpiece WK is a model that represents the shape of the workpiece WK used when performing the search process described later. When registering the search model of the workpiece WK in step SB1, three-dimensional CAD data (CAD model) of the workpiece WK is read and temporarily stored in the storage device 42 of the control device 100. The CAD data is data of the search model of the workpiece WK, and data in a format that has been commonly used in the past can be used.

The search model for the workpiece WK may be, for example, an interference detection model (used in the interference detection unit 25 described later) generated based on the surface shape seen from a certain side. The search model for the workpiece WK may be in the STL format, which is the simplest format for CAD data.

After loading the CAD data of the search model of the workpiece WK in step SB1, proceed to step SB2 to confirm and register the search model of the workpiece WK.

In addition, in step SB2, height image data viewed from each direction of the six faces of the CAD model read in step SB1 is generated. First, six height image data are generated so as to provide a plan view, bottom view, left side view, right side view, front view, and back view of the CAD model. A height image is obtained from this height image data. If there is no CAD data for the workpiece WK, multiple height image data viewed from different directions of the search model of the workpiece WK may be registered as a search model. That is, the workpiece WK is placed on a plane with the face to be registered facing up, and three-dimensional measurement is performed. This three-dimensional measurement can be performed using the three-dimensional camera 2 and shape measurement processing unit 200 of the robot system 1000. The measurement data obtained by the three-dimensional measurement by the shape measurement processing unit 200 is output from the shape measurement processing unit 200, and height image data of the face to be registered of the workpiece WK can be obtained based on the measurement data. After the height image is obtained, the height image is registered as a search model.

In step SB3 of the flowchart shown in FIG. 5, search conditions are set. The search conditions include, for example, the number of workpieces WK that can be detected during a search, a lower limit of the correlation value between the search model and the workpieces WK, feature extraction conditions (edge extraction threshold value, etc.), etc. The search conditions are set by the user and are accepted by the setting unit 21.

Next, the flowchart showing the procedure for setting the hand model and gripping position shown in FIG. 6 will be described. In step SC1 after the start, a robot model that specifies various information related to the robot RBT to be used at the site is registered.

After registering the robot model in step SC1, proceed to step SC2 to register the robot hand model. The robot hand model is a model that represents the shape of the robot hand, and for example, polygon data (CAD data) of the robot hand is read in.

Then, proceed to step SC3, where you define the relative position and orientation between the robot hand model registered in step SC2 and the flange surface of the robot model (already registered in step SC1) to which the robot hand model will be attached. Specifically, you define the positional relationship between the mounting surface of the robot hand model and the flange surface of the robot model so that they coincide with each other.

In step SC4 following step SC3, the tip coordinates of the robot hand model attached to the flange surface of the robot model are specified. These tip coordinates can be obtained from the robot controller 6, but can also be set by the user. Then, in step SC5, it is confirmed that the robot hand model has been registered.

Then, in step SC6, a surface of the search model to be grasped by the hand model is selected. The surface of the search model can be represented by a height image registered in the flowchart shown in FIG. 5, and the user selects a height image corresponding to a surface that can be grasped by the robot hand model from among the multiple registered height images.

After selecting the height image in step SC6, proceed to step SC7 and register the position and posture of the robot hand HND when gripping the surface selected in step SC6. The gripping information indicating the candidate gripping positions of the workpiece WK and the posture of the robot hand HND can be stored in the gripping information storage unit 42a (shown in FIG. 2) of the storage device 42 of the control device 100 in a state associated with the search model.

When picking up the workpiece WK with the actual robot hand HND, it is necessary to convert the vision coordinates, which are the coordinates of the three-dimensional space (vision space) in which the workpiece WK is imaged by the three-dimensional camera 2, into the robot coordinates that the robot controller 6 uses when actually operating the robot RBT. The coordinates can be given in robot coordinates (XYZ, RxRyRz), but they can also be given in the axis angle (joint angle) of the arm section ARM. In addition, the axis angle can be calculated from the robot coordinates of the gripping position, etc. within the control device 100, and the axis angle can be transmitted to the robot controller 6.

Specifically, the position and orientation of the search model of the workpiece WK are obtained as a position and orientation in the vision space. Similarly, the orientation of the robot hand model that grasps it is obtained as a position and orientation in the virtual three-dimensional space of the control device 100. In order for the robot controller 6 to operate the robot RBT based on the position and orientation in this vision space, these must be converted into a position and orientation in the robot space. The process of calculating a conversion equation for converting the position and orientation calculated in this displayed coordinate system into the position and orientation in the coordinate system in which the robot controller 6 operates the robot hand HND can be performed by a conventionally known method.

In step SC8 of the flowchart shown in FIG. 6, it is determined whether the required number of gripping candidate positions have been registered. If the determination in step SC8 is NO and the required number of gripping candidate positions have not been registered, the process proceeds to step SC7 via step SC6, where other gripping candidate positions are set and registered. On the other hand, if the determination in step SC8 is YES, the process proceeds to step SC9.

In step SC9, the start position and the placement position where the grasped workpiece WK is placed are registered. The start position here is the position where the robot hand HND grasps the workpiece WK, and corresponds to the transfer start position. The start position can be a candidate gripping position for the workpiece WK bulk-stacked in the container. The start position will differ depending on where the workpiece WK is located in the container, but this can be obtained by a three-dimensional search. On the other hand, the placement position is the position where the workpiece WK is placed by the robot hand HND, and can be set, for example, on the stage STG shown in FIG. 1.

In step SC10, the waypoints through which the workpiece WK will be moved from the gripping position to the placement position are registered. When the waypoints are registered, the operation of the arm unit ARM and the robot hand HND is set so that part of the robot hand HND or part of the workpiece WK passes through the waypoints while the robot hand HND grips the workpiece WK and transports it from the starting position to the placement position.

(Basic robot control)
7 is a diagram showing an example of a flowchart of the basic control of the robot RBT. In step SD1 after starting, the measurement unit 20 captures an image of the workpiece WK to measure the position and orientation of the workpiece WK, and generates data for search processing (height image data) executed in step SD2. Since the measurement unit 20 is capable of three-dimensional measurement, the data for search processing can include height information.

Then, proceed to step SD2, and execute a search process to detect the position and orientation of the workpiece WK from the search process data generated in step SD1. Specifically, the registered search model of the workpiece WK is used to identify the position and orientation of the workpiece WK. First, the search process data is searched for the position and orientation that best matches each feature point of the search model.

If, as a result of searching to see whether a registered search model exists, the search model does not exist in the search processing data, i.e., the work WK cannot be detected, this flowchart ends.

In addition, in step SD2, the position determination processing unit 22 shown in FIG. 2 determines a gripping position of the workpiece WK arranged in the workspace by the robot hand HND based on the measurement results by the measurement unit 20 and the gripping information stored in the gripping information storage unit 42a. Furthermore, after determining the gripping position, the position determination processing unit 22 is configured to calculate a movement target coordinate of the robot hand HND for gripping the gripping position.

At this time, the judgment result of the interference judgment unit 25 shown in FIG. 2 can be used. The interference judgment unit 25 is a part that judges interference between the robot hand HND and an obstacle. For example, it can judge whether or not the model of the robot hand HND registered in the process shown in the flowchart in FIG. 6 will come into contact with an obstacle such as the wall of the container BX.

In addition, in step SD3, the path generation unit 23 shown in FIG. 2 generates a transport path for transporting the workpiece WK grasped by the robot hand HND to the placement position. In step SD4, the output unit 24 outputs the coordinates of the gripping position, placement position, waypoint, etc. and the movement target coordinates calculated in step SD2 to the robot controller 6. The robot controller 6 receives the coordinates of the gripping position, placement position, waypoint, etc. output from the output unit 24, generates a control signal, and outputs the generated control signal to the robot RBT. After that, in step SD5, the robot RBT operates in accordance with the control signal to transport the workpiece WK along a predetermined path.

(calibration)
The robot system 1000 can be operated according to the flowchart shown in FIG. 7. However, because the robot system 1000 is a hand-eye type robot system, before operation, it is necessary to perform calibration, known as hand-eye calibration, for converting the camera coordinates of the three-dimensional camera 2 into the robot coordinates of the robot hand HND.

When performing calibration, calibration marks and a calibration board may be used as disclosed in the above-mentioned Patent Document 1. However, since the calibration marks and the calibration board differ from the workpiece WK in terms of material, shape, size, etc., it is necessary to change the imaging parameters and the focal position of the three-dimensional camera 2 from those used when imaging the actual workpiece WK when imaging the calibration marks and the calibration board. In other words, since calibration is performed with imaging parameters that are different from those used when the robot system 1000 is in operation, it is difficult to say that a calibration result that is optimal for operation can be obtained.

Therefore, in this embodiment, the workpiece WK that is actually picked during operation is used for calibration. The control device 100 is configured to measure the shape of the workpiece WK using a three-dimensional camera 2 fixed to the robot hand HND, and to perform multiple stages of calibration based on the measured shape data of the workpiece WK.

The calibration procedure is roughly divided into the first to third steps SE1 to SE3 as shown in FIG. 8. In the first step SE1 after starting, the robot hand HND and the three-dimensional camera 2 are translated in the XYZ directions, the camera coordinates and the robot coordinates are obtained when the movement of the robot hand HND and the three-dimensional camera 2 is stopped, and the correspondence between the camera coordinates and the robot coordinates is obtained. In the second step SE2, the robot hand HND and the three-dimensional camera 2 are rotated multiple times around a predetermined axis, the camera coordinates and the robot coordinates are obtained when the movement of the robot hand HND and the three-dimensional camera 2 is stopped, and a provisional correspondence between the rotation center and the position of the three-dimensional camera 2 is obtained. In the third step SE3, the robot hand HND and the three-dimensional camera 2 are rotated based on the provisional correspondence obtained in the second step SE2, and a correction value for the rotation center is obtained. Based on the obtained correction value, the provisional rotation center obtained in the second step SE2 is corrected.

In the flowchart shown in FIG. 8, the first step SE1 and the second step SE2 are shown as separate steps, but this is not limiting, and the processing content in the first step SE1 and the processing content in the second step SE2 may be executed in a single step. Also, in the second step SE2, a provisional correspondence relationship between the center of rotation and the position of the three-dimensional camera 2 is obtained, but if this relationship is predetermined by a design value, the second step SE2 can be omitted and substituted by reading in the design value.

The first step SE1 and the second step SE2 are defined as normal calibration processes because they are steps for calculating parameters for converting a position in the camera coordinates to a position in the robot coordinates and for determining a provisional correspondence between the center of rotation and the position of the three-dimensional camera 2. The third step SE3 is defined as high-precision calibration processes because it is a step for correcting the provisional center of rotation determined in the second step SE2 to improve accuracy. The normal calibration process and the high-precision calibration process are performed by the calibration execution unit 26 shown in FIG. 2.

The control device 100 is configured to be able to execute an operation mode selected when operating the robot system 1000 described above, and a calibration mode selected when performing calibration. When a user selects a calibration mode for performing calibration before operating the robot system 1000, the calibration execution unit 26 detects that the calibration mode has been selected in the control device 100. When the calibration execution unit 26 detects that the calibration mode has been selected, it executes the normal calibration process and the high-precision calibration process in sequence and continuously. "Executing continuously" means that after the normal calibration process is completed, the high-precision calibration process is started using the result of the normal calibration process without the user having to perform any special operation, thereby reducing the burden on the user. Note that the high-precision calibration process may be started after the normal calibration process is completed, waiting for an instruction from the user, without executing the normal calibration process and the high-precision calibration process continuously. Details of each of the first to third steps SE1 to SE3 are described below.

(Details of the first step)
9 is a flowchart showing a detailed procedure of the first step SE1 shown in FIG. 8. When the first step SE1 is started, the calibration execution unit 26 generates a calibration user interface 50 consisting of an image as shown in FIG. 10 and displays it on the display unit 3. The calibration user interface 50 is provided with a two-dimensional image display area 51 that displays a color image of the workpiece WK captured by the three-dimensional camera 2, and a three-dimensional image display area 52 that displays a height image based on three-dimensional shape data acquired from the image captured by the three-dimensional camera 2. In reality, the three-dimensional image display area 52 displays a workpiece image colored in different colors depending on the height, but coloring is omitted in the figure of this example.

The calibration user interface 50 further includes a mode display area 53 that displays whether or not the calibration mode is selected, a movement amount display area 54 that displays the movement amount of the robot hand HND, an error display area 55 that displays the rotation center error, and a movement amount setting area 56 that sets the movement amount of the robot hand HND during normal calibration processing. The movement amount display area 54 displays the movement amount (X) of the robot hand HND in the X direction, the movement amount (Y) of the Y direction, the movement amount (Z) of the robot hand HND, the rotation amount (Rx) around the X axis, the rotation amount (Ry) around the Y axis, and the rotation amount (Rz) around the Z axis. The units of the movement amount (X) in the X direction, the movement amount (Y) in the Y direction, and the movement amount (Z) in the Z direction are mm, and the units of the rotation amount (Rx) around the X axis, the rotation amount (Ry) around the Y axis, and the rotation amount (Rz) around the Z axis are deg.

The movement amount setting area 56 also displays the amount of movement in the X direction (X), the amount of movement in the Y direction (Y), the amount of movement in the Z direction (Z), the amount of rotation around the X axis (Rx), the amount of rotation around the Y axis (Ry), and the amount of rotation around the Z axis (Rz) similar to the movement amount display area 54, and the user can input each value individually. Each value in the movement amount setting area 56 may be a predetermined value input, or the input value may be changeable by the user. The user can input or change each value by operating the operation unit 4.

In step SF1 of the flowchart shown in FIG. 9, the robot hand HND is moved to the start position of the calibration. After being moved to the start position, the process proceeds to step SF2, where the robot hand HND is translated in parallel in a grid pattern of 3 vertical points x 3 horizontal points x 3 vertical points (total of 27 points) with the start position as the center, as shown in FIG. 11. The amount of movement at this time is the amount of movement in the X direction (X), the amount of movement in the Y direction (Y), and the amount of movement in the Z direction (Z) input in the movement amount setting area 56 of the calibration user interface 50 shown in FIG. 10.

In this step SF1, the robot hand HND is not rotated, but is moved only in the directions along the X, Y, and Z directions. In other words, the posture (θx, θy, θz) of the robot hand HND is not changed, and θx, θy, and θz are kept constant. This is to avoid breaking the relationship between the tool center and the position of the three-dimensional camera 2. Note that the vertical direction is the Y direction, the horizontal direction is the X direction, and the height is the Z direction. The intervals between multiple points in the X direction, the intervals between multiple points in the Y direction, and the intervals between multiple points in the Z direction can all be the same. In addition, in this embodiment, a total of 27 points are moved, but this is just one example, and the number of moving points may be less than or greater than 27. In addition, the robot hand HND does not necessarily have to be moved in the directions along the X, Y, and Z directions, and may be moved linearly with a predetermined inclination angle with respect to the X direction, may be moved linearly with a predetermined inclination angle with respect to the XY direction, or may be moved linearly with a predetermined inclination angle with respect to the Z direction.

After each operation, the workpiece WK is imaged by the three-dimensional camera 2 at the stop position of the robot hand HND. FIG. 12 is a diagram showing an example of a calibration user interface 50 displayed when the robot hand HND is moved to the position of the first point. FIG. 13 is a diagram showing an example of a calibration user interface 50 displayed when the robot hand HND is moved to the position of the 27th point. Since the position of the three-dimensional camera 2 relative to the workpiece WK is different between the first point and the 27th point, the three-dimensional image display area 52 in FIG. 12 and the three-dimensional image display area 52 shown in FIG. 13 display three-dimensional images of the workpiece WK viewed from different angles and positions. The three-dimensional images of the workpiece WK imaged at each of the first point to the 27th point may be displayed sequentially in the three-dimensional image display area 52, or the three-dimensional image of the workpiece WK may not be displayed. As shown in FIG. 12 and FIG. 13, when imaging is started at each of the first point to the 27th point, an option display area 57 is displayed in the calibration user interface 50. The option display area 57 displays position tolerances, normal tolerances, angle tolerances, etc. Note that tolerances are not relevant during calibration, but are applied during operation.

The calibration execution unit 26 detects the work coordinates based on the three-dimensional shape data obtained by measurement using the three-dimensional camera 2, and obtains the robot coordinates corresponding to the work coordinates. The robot coordinates corresponding to the work coordinates are the robot coordinates when the work coordinates are obtained. This is performed for all 27 points, and the work coordinates and robot coordinates for each point are stored as pairs (27 pairs) in the storage device 42, etc.

Methods for identifying the position of the workpiece WK at each point include, for example, a method of searching the three-dimensional shape captured by the three-dimensional camera 2 and using the XYZ coordinates of its representative point (such as the center position) as the detection coordinates, and a method of searching the two-dimensional shape and using the XY coordinates of the representative point as the representative point and calculating the Z of the representative point from a separately calculated plane equation. In this embodiment, the latter identification method is adopted, and after detecting the edges of multiple points, the center coordinates of the circle are calculated using the least squares method so as to follow the changes in the appearance of the workpiece WK depending on the angle of view. The coordinates of the representative point may be found as the coordinates of a characteristic part of the workpiece WK. For example, if there is a hole in the center of the workpiece WK, the center of the hole may be used as the representative point.

Then, proceed to step SF3 in the flowchart shown in FIG. 9. In step SF3, a correction process is performed to correct the angle of view of the three-dimensional camera 2. If the angle of view of the three-dimensional camera 2 has been corrected, this step SF3 can be omitted. Furthermore, the three-dimensional camera 2 used in this embodiment is of an interchangeable lens type, and is configured so that angle of view correction is not performed on the camera side, but it may be configured so that angle of view correction is performed on the camera side or within the control device 100 when the lens is replaced. A camera with a non-interchangeable lens may also be used in this embodiment.

Here, we will explain how the workpiece WK appears in the field of view when the workpiece WK is imaged by the three-dimensional camera 2. In step SF1 above, the robot hand HND to which the three-dimensional camera 2 is fixed is translated in the vertical, horizontal, and height directions, so the position of the workpiece WK shifts relative to the field of view of the three-dimensional camera 2. In the case of a three-dimensional camera 2 that does not correct the angle of view, if the distance from the three-dimensional camera 2 changes, the area of the workpiece WK that appears in the field of view will also change. In a three-dimensional camera 2 that does not correct the angle of view, the Z direction can be said to be in mm units and the X and Y directions in pixel units, and in a three-dimensional camera 2 that does correct the angle of view, the X and Y directions can also be said to be in mm units.

This will be explained based on Figures 14A and 14B. Figure 14A is a side view showing the possible positional relationship between the workpiece WK and the three-dimensional camera 2 when measuring points 1 to 27 in the first step. Plane S1 is a plane extending horizontally from the top surface of the workpiece WK. Plane S2 is a horizontal plane passing through the light receiving element (or the light receiving surface of the lens) when the three-dimensional camera 2 is in the reference position, i.e., the middle position (first position). Plane S3 is a horizontal plane passing through the light receiving element (or the light receiving surface of the lens) when the three-dimensional camera 2 is in the upper position (second position). If the representative point of the workpiece WK is point P1, the physical coordinates of point P1 can be expressed as (X, Z) at the reference position and (X, Z') at the upper position.

The straight line L1 indicates the optical axis of the three-dimensional camera 2 and is aligned along the Z direction. The two straight lines L2, L2 indicate the field of view of the three-dimensional camera 2 when the three-dimensional camera 2 is in the reference position. Furthermore, the two straight lines L3, L3 indicate the field of view of the three-dimensional camera 2 when the three-dimensional camera 2 is in the upper position.

The rectangle indicated by the symbol 2A in Figure 14B indicates the field of view range when imaged by the three-dimensional camera 2. The workpiece WK imaged when the three-dimensional camera 2 is in the reference position is indicated by a solid line, and the workpiece WK imaged when the three-dimensional camera 2 is in the upper position is indicated by a dashed line. The coordinates of the representative point P1 of the workpiece WK imaged when the three-dimensional camera 2 is in the reference position are (xp, Z), and the coordinates of the representative point P1 of the workpiece WK imaged when the three-dimensional camera 2 is in the upper position are (xp', Z').

The relationship shown in FIG. 14A and FIG. 14B can be expressed in a table as shown in FIG. 15. That is, the horizontal distance from the center (optical axis L1) of the three-dimensional camera 2 to the representative point P1 of the workpiece WK is xp when it is in the middle position and xp' when it is in the upper position when it is in the upper position when it is in the middle position when it is in the upper position when it is in the upper position when it is in the upper position, and these xp and xp' are known because they are detected representative points. Also, the vertical distance (distance in the Z direction) from the three-dimensional camera 2 to the representative point P1 of the workpiece WK is Z when it is in the middle position and Z' when it is in the upper position when it is in the upper position when it is in the middle position when it is in the upper position, and these Z and Z' are known because they are detected representative points. Of the horizontal distances from the center (optical axis L1) of the three-dimensional camera 2 to the representative point P1 of the workpiece WK, the physical distance X is unknown, but the geometric relationship Z':Z = xp:xp' holds.

When the three-dimensional camera 2 is in the upper position, the pixel position xp' is calculated from the relationship between the physical size (position) X and the distance Z' from the three-dimensional camera 2 depending on the angle of view. However, if this is left as it is, the camera coordinates themselves depend on the height, making it difficult to use for calibration. Therefore, the calibration execution unit 26 of this embodiment converts the above relationship into a pixel position that is independent of the camera's angle of view (i.e., when a telecentric lens is used). xcp and X are in a proportional relationship, that is, the corrected pixel position xcp depends on X. From this relationship, xcp can be calculated using the following formula 1 (a formula for calculating the equivalent pixel position on the reference surface). Note that, when considering lens distortion, other calculation methods can also be adopted, so the calculation method is not limited to the method using formula 1.

xcp=xp’*(Z’/Z) … Formula 1

The above is the correction process executed in step SF3 of the flowchart shown in FIG. 9. Then, proceed to step SF4, where the correspondence from the camera coordinates to the robot coordinates is found as an affine transformation matrix using the least squares method. Specifically, in this example, the workpiece WK is measured at 27 points, so the affine transformation matrix can be found using the least squares method from the relationship between the 27 robot positions Pr and the camera coordinate position Pc. Here, Pc is the corrected camera position. The corrected camera position Pc does not need to have X, Y and Z in mm units. In this example, X and Y are pixel unit coordinates, and Z is mm coordinates, and as long as the X, Y and Z coordinates are in a uniform space, they can be used for calculation.

Through step SF4, the first step SE1 of the flowchart shown in FIG. 8 is completed. FIG. 16 shows a calibration user interface 50 that is displayed on the display unit 3 when the first step SE1 is completed. This calibration user interface 50 is provided with a calibration result display area 58. The calibration result display area 58 displays the field of view, pixel resolution, coordinate axis angles, coordinate axis rotation angles, rotation center position, etc. of the three-dimensional camera 2 determined in the first step SE1.

(Details of the second step)
Fig. 17 is a flow chart showing the detailed procedure of the second step SE2 shown in Fig. 8. When the second step SE2 is started, the calibration execution unit 26 generates a calibration user interface 50 shown in Fig. 18 and displays it on the display unit 3. The calibration user interface 50 displayed in the second step SE2 displays in the mode display area 53 that it is in the rotation center calculation mode.

In step SG1 of the flowchart shown in FIG. 17, the robot hand HND is moved to the start position of the calibration. After being moved to the start position, the process proceeds to step SG2, where the robot hand HND is stopped at each point (seven points in total) that is shifted by the start position, ±θx, ±θy, and ±θz relative to the tool center coordinates, and the workpiece WK is imaged by the three-dimensional camera 2 at the stop position of the robot hand HND. The workpiece coordinates are detected based on the three-dimensional shape data obtained by measurement with the three-dimensional camera 2, and the robot coordinates corresponding to the workpiece coordinates are obtained. The workpiece coordinates and robot coordinates of each point are paired and stored in the storage device 42 or the like. After that, the process proceeds to step SG3, where a provisional center of rotation is determined based on the robot coordinates.

Specifically, when finding the center of rotation from the detected coordinates of seven points, the least squares method using the equation of a sphere can be used. In this example, the number of points is seven, but it is possible to find the center of rotation in three dimensions if there are four or more points. The more points there are, the better the results will be.

The center of rotation can also be found geometrically without using the equation of a sphere. For example, multiple bisectors and bisectors of the displacement vector can be found, and the center of rotation can be estimated from their intersection point. The distance to the center can be found using arctan from the relationship between the displacement distance and the angle of the displacement, and the central coordinates can be found from these multiple relationships.

The above is the second calibration step SE2, but in principle, it is difficult to accurately determine the center of rotation unless the rotational movement of the robot hand HND is large. However, as shown in FIG. 19, if the rotational movement of the robot hand HND is large, the workpiece WK does not fit into the field of view, and detection of the representative point P1 becomes unstable. The three symbols 2A in FIG. 19 indicate the field of view of the three-dimensional camera 2, with the right field of view 2A indicating the field of view before rotation, and the center field of view 2A and the left field of view 2A indicating the field of view after rotation.

When the second step is completed, a calibration user interface 50 as shown in FIG. 20 is displayed, and the third step SE3 shown in FIG. 8 above is started.

(Details of the third step)
Fig. 21 is a flow chart showing a detailed procedure of the third step SE3 shown in Fig. 8. In the third step SE3, the robot hand HND is moved by a movement amount set in a high-precision calibration setting area 60 provided in the calibration user interface 50 shown in Fig. 22. The calibration user interface 50 shown in Fig. 22 may be displayed on the display unit 3 before the start of the first step SE1 so that each value can be input, or may be displayed on the display unit 3 after the second step SE2 is completed so that each value can be input. The movement amount set in the high-precision calibration setting area 60 may be a predetermined value.

In step SH1 of the flowchart shown in FIG. 21, a provisional rotation center (provisional rotation center) is determined based on the relative position of the tool center of the robot hand HND and the three-dimensional camera 2 obtained by the normal calibration process. The determined provisional rotation center is stored in the storage device 42 or the like as the original rotation center. Then, the process proceeds to step SH2, where the robot hand HND is moved to a position (start position) where the representative point P1 of the workpiece WK can be stably obtained.

Next, proceed to step SH3, and using the relationship between the camera position and the original rotation center position, rotate the robot hand HND (three-dimensional camera 3) at the starting position, stop the robot hand HND at each point (seven points in total) shifted from the starting position by ±θx, ±θy, ±θz, and capture an image of the workpiece WK with the three-dimensional camera 2 at the stopping position of the robot hand HND. Based on the three-dimensional shape data obtained by measurement with the three-dimensional camera 2, the workpiece coordinates (coordinates of representative point P1) are detected, and the robot coordinates corresponding to the workpiece coordinates are obtained. The workpiece coordinates and robot coordinates of each point are paired and stored in a storage device 42, etc.

When setting the rotation angle, the calibration execution unit 26 sets the rotation angle (θx, θy, θz) of the robot hand HND rotated around the temporary rotation center in the high-precision calibration process to a rotation angle greater than the rotation angle of the robot hand HND rotated around the XYZ axes in the normal calibration process. Note that the rotation angle of the robot hand HND may be the same in the high-precision calibration process and the normal calibration process.

In step SH3, the provisional positional relationship between the tool center and the camera position is obtained by the normal calibration process, and the robot hand HND can be rotated while the workpiece WK is in the field of view so that the representative point P1 of the workpiece WK can be stably obtained by using the provisional positional relationship. That is, as shown in FIG. 23A, if the robot hand HND is simply rotated around the provisional rotation center, a part of the workpiece WK will be out of the field of view 2A after the rotation, and as a result, the representative point P1 will be located outside the field of view 2A and will not be measurable. On the other hand, as shown in FIG. 23B, in this high-precision calibration process, the calibration execution unit 26 rotates the robot hand HND around the provisional rotation center by a predetermined angle and moves the robot hand HND so that the workpiece WK after the predetermined angle rotation is in the field of view 2A of the three-dimensional camera 2, so that the entire workpiece WK is located within the field of view 2A, and as a result, the representative point P1 can be positioned within the field of view 2A.

In other words, the expected amount of movement when rotated a specified angle is calculated based on the original center of rotation, and the robot hand HND is moved or rotated using this amount of movement to capture an image with the three-dimensional camera 2. From the representative point found from this, the position of the representative point that was originally desired can be found. Specifically, since a provisional center of rotation is found in the normal calibration process and the position after movement due to rotation can be estimated, the movement of the camera position can be offset by controlling the tool center to move in the opposite direction of that movement vector. This makes it possible to rotate the robot hand HND by a large amount while stably detecting the representative point of the workpiece WK.

For example, if there is a deviation between the original rotation center and the true rotation center position as shown in Figure 24, the deviation can be found as follows. Note that p, c, and d are vectors, and Q is a rotation matrix. If the origin rotation center is c and the detected deviation of the representative point p is d, the following relationship holds.

Q(pc)+d+c...Formula 2
Q(p-(c+x))+(c+x)...Equation 3

Since Equation 2 and Equation 3 are equivalent, the center deviation x and the true center of rotation c+x can be calculated. Note that I is a unit vector.

d = -Qx + x
d = (I-Q) x
x=(I-Q)^-1d

In the case of two dimensions, the inverse matrix (I-Q) can be found reliably. In the case of three dimensions, the above relationship alone will result in a rank loss, so the number of equations is increased and the solution is found using the least squares method. To achieve this, θx, θy, and θz are each deflected in the ± direction.

Then, proceed to steps SH4 to SH6 of the flowchart shown in FIG. 21, and determine the final rotation center of the robot hand HND based on the pair of work coordinates and robot coordinates of each point held in step SH3. To be more specific, first, in step SH4, the rotation center is recalculated (corrected) based on the pair of work coordinates and robot coordinates of each point held in step SH3. The recalculated value is saved, and this value is overwritten as the next original rotation center. Steps SH2 to SH4 are repeated N times, and the value of the rotation center for each loop is saved. For example, the first loop can be called the first process, and the second loop can be called the second process, and the calibration execution unit 26 can determine the final rotation center of the robot hand HND after at least the first process and the second process.

After steps SH2 to SH4 are repeated N times, proceed to step SH5. In step SH5, the average of the N rotation center values is calculated. Rotation centers that deviate from the average by a predetermined threshold or more are marked in order to remove outliers.

Then, proceed to step SH6, calculate the average using only the rotation centers that were not marked in step SH5, and use the resulting average as the final rotation center.

In this flow chart, the original rotation center is updated for each loop (a loop consisting of steps SH2 to SH4). A conceptual diagram of updating the original rotation center for each loop is shown in FIG. 25. In FIG. 25, the initial original rotation center is the provisional rotation center obtained by the normal calibration process, and as shown by the arrow, the rotation center is updated for each loop. As shown in this example, the rotation center may deviate significantly from the average. If the rotation center deviates significantly, it can be estimated that the rotation center is an abnormal value, so when updating the original rotation center, a process of removing the abnormal value may be performed. Specifically, in this flow, after calculating N rotation centers by going through the above loop N times, those calculated rotation centers that deviate from the average by a predetermined threshold value or more are eliminated. Alternatively, a threshold value can be subtracted from the correction amount for each loop to eliminate rotation centers with a large error. This process can also be performed by the calibration execution unit 26.

When the original rotation center is updated for each loop, the obtained rotation center gradually approaches the true rotation center, and the obtained rotation center converges. In this embodiment, for example, a convergence value display area 62 is provided in the calibration user interface 50 shown in FIG. 20. The convergence value display area 62 can display the state in which the rotation center converges. As an example of the display, the convergence value (unit: mm) of the rotation center is displayed. For example, the convergence value of the rotation center obtained in the third loop is displayed as a smaller value than the convergence value of the rotation center obtained in the second loop, and the convergence value displayed in the convergence value display area 62 becomes smaller with each increase in the number of loops. Therefore, by looking at the convergence value displayed in the convergence value display area 62, the user can understand that the obtained rotation center is gradually approaching the true rotation center. The number of loops may be two.

If the convergence value does not change by a predetermined amount even if the number of loops is increased, the high-precision calibration process can be terminated at that point. Specifically, the calibration execution unit 26 determines whether the multiple rotation centers acquired by repeatedly executing the high-precision calibration process satisfy a predetermined convergence condition. The calibration execution unit 26 can be configured to terminate the high-precision calibration process when it is determined that the predetermined convergence condition is satisfied, and to continue the high-precision calibration process when it is determined that the predetermined convergence condition is not satisfied. An example of the predetermined convergence condition is that the change in the error of the rotation center acquired by the high-precision calibration process is equal to or less than a certain amount. If the change in the error of the rotation center is equal to or less than a certain amount for multiple consecutive times (for example, three times), it may be determined that the predetermined convergence condition is satisfied. The predetermined convergence condition may be determined in advance, or may be set arbitrarily by the user.

When the high-precision calibration process is completed and the final center of rotation is determined, the calibration execution unit 26 generates a calibration user interface 50 shown in FIG. 26 and displays it on the display unit 3, while notifying the user that the high-precision calibration process is completed. The calibration user interface 50 displayed after the high-precision calibration process is completed has a calibration result display area 61. The calibration result display area 61 displays the field of view, pixel resolution, coordinate axis angles, coordinate axis rotation angles, final rotation center position, etc. of the three-dimensional camera 2 acquired through both the normal calibration process and the high-precision calibration process.

(Effects of the embodiment)
As described above, according to this embodiment, the three-dimensional camera 2 is fixed to the robot hand HND, so that when the robot system 1000 is in operation, it is possible to measure the position of the workpiece WK using the three-dimensional camera 2 without significantly retracting the robot hand HND, and it is also possible to measure the three-dimensional shape of the workpiece WK to be measured from a location close to the workpiece WK, thereby improving measurement accuracy.

In addition, during calibration, the workpiece WK can be used without using a calibration mark or a calibration board. At this time, since the parameters for converting the position in the camera coordinates to the position on the robot coordinates can be calculated by the normal calibration process first, the relative positional relationship between the tool center of the robot hand HND and the three-dimensional camera 2 can be obtained. After that, the provisional rotation center of the robot hand HND is determined, and the robot hand HND is rotated around the provisional rotation center multiple times, so that the workpiece WK does not go out of the field of view of the three-dimensional camera 2, the entire workpiece WK is placed in the field of view of the three-dimensional camera 2, and the workpiece WK is placed near the center of the field of view of the three-dimensional camera 2. Then, at each position, the workpiece WK is measured by the three-dimensional camera 2 to detect the workpiece coordinates, and the robot coordinates when the workpiece coordinates are obtained are obtained, and the final rotation center of the robot hand HND can be determined with high accuracy based on the obtained pairs of the multiple workpiece coordinates and the robot coordinates.

The above-described embodiments are merely illustrative in all respects and should not be interpreted as limiting. Furthermore, all modifications and variations within the scope of the claims are within the scope of the present invention.

As described above, the control device according to the present invention can be applied to a robot system.

2 Three-dimensional camera 3 Display unit 26 Calibration execution unit 50 Calibration user interface 100 Control device HND Robot hand RBT Robot WK Workpiece

Claims (10)

A control device that measures a shape of a workpiece using a three-dimensional camera fixed to a robot hand, and performs calibration to convert camera coordinates of the three-dimensional camera into robot coordinates of the robot hand based on measured shape data of the workpiece,
a first normal calibration process for executing an operation of moving the robot hand in X, Y and Z directions, detecting workpiece coordinates based on shape data obtained by measuring the workpiece with the three-dimensional camera at a stop position of the robot hand, acquiring robot coordinates at the time when the workpiece coordinates are obtained, and calculating parameters for converting a position in the camera coordinates to a position on the robot coordinates based on a plurality of pairs of the acquired workpiece coordinates and robot coordinates;
a second normal calibration process for executing an operation of rotating the robot hand around a predetermined axis, detecting workpiece coordinates based on shape data obtained by measuring the workpiece with the three-dimensional camera rotated in conjunction with the rotation of the robot hand at a stop position of the robot hand, acquiring robot coordinates at the time when the workpiece coordinates were obtained, and determining a provisional rotation center of the robot hand based on the robot coordinates based on the acquired pairs of multiple workpiece coordinates and robot coordinates;
a high-precision calibration process in which the provisional rotation center is set as the original rotation center of the robot hand, an operation is performed to rotate the robot hand around the original rotation center, work coordinates are detected based on shape data obtained by measuring the work at the stop position of the robot hand using the three-dimensional camera that rotates in conjunction with the rotation of the robot hand, the robot coordinates at the time the work coordinates are obtained are obtained, a next rotation center is determined based on the obtained pairs of multiple work coordinates and robot coordinates, the original rotation center of the robot hand is updated to the next rotation center, and the operation of rotating the robot hand around each updated original rotation center is repeated to determine a final rotation center of the robot hand;
A calibration execution unit that executes the following :
The calibration execution unit is a control device that, during the high-precision calibration process, determines the next rotation center corresponding to the original rotation center based on the deviation between the work coordinates after the robot hand has rotated, which are estimated based on the original rotation center, and the work coordinates after the robot hand has rotated, which are detected based on measurements by the three-dimensional camera. 2. The control device according to claim 1,
The calibration execution unit is a control device that sets the rotation angle of the robot hand rotated around the original rotation center in the high-precision calibration process to a rotation angle larger than the rotation angle of the robot hand rotated around the specified axis in the second normal calibration process. 3. The control device according to claim 1 ,
The calibration execution unit repeatedly executes a process of determining a next rotation center corresponding to each of the updated original rotation centers to obtain multiple rotation centers of the robot hand, and determines the final rotation center based on the multiple rotation centers obtained. 4. The control device according to claim 3 ,
The calibration execution unit determines whether the multiple rotation centers obtained by executing the repetition satisfy a predetermined convergence condition, and terminates the repetition when it is determined that the predetermined convergence condition is satisfied. 5. The control device according to claim 1,
The calibration execution unit is a control device that rotates the robot hand a predetermined angle around the original rotation center during the high-precision calibration process and moves the robot hand so that the workpiece after rotation by the predetermined angle is within the field of view of the three-dimensional camera. 6. The control device according to claim 1,
The calibration execution unit is a control device that calculates the XYZ coordinates of a representative point of the work based on three-dimensional shape data obtained by measuring the work using the three-dimensional camera, and defines the calculated XYZ coordinates of the representative point as work coordinates. 7. The control device according to claim 1,
The calibration execution unit is a control device that calculates the XY coordinates of a representative point of the work based on two-dimensional shape data obtained by measuring the work using the three-dimensional camera, and calculates the Z coordinate of the representative point from a planar equation, and sets the calculated XYZ coordinates of the representative point as work coordinates. 8. The control device according to claim 1,
The calibration execution unit detects work coordinates based on shape data obtained by moving the robot hand to a first position and measuring the work with the three-dimensional camera, and detects work coordinates based on shape data obtained by moving the robot hand to a second position having a Z coordinate different from the first position and measuring the work with the three-dimensional camera, and converts the work coordinates detected at the first position and the work coordinates detected at the second position into pixel positions independent of the angle of view of the three-dimensional camera. 9. The control device according to claim 1,
It is configured to allow selection of the calibration mode in which calibration is performed,
The calibration execution unit is a control device that, when detecting that the calibration mode has been selected, successively executes the first normal calibration process, the second normal calibration process, and the high-precision calibration process in that order. 10. The control device according to claim 1,
The calibration execution unit generates a user interface having a display area capable of updating and displaying the detection results of the work coordinates detected in the high-precision calibration process and the convergence value of the rotation center, and is equipped with a display unit that displays the user interface generated by the calibration execution unit.

Images