JP7797664B2 - Robot system having a robot and method for calibrating the robot - Google Patents

Description

The present disclosure relates generally to the field of robotics, and more specifically to a robotic system having a robot and a method for calibrating a robot.

In today's world, technology is growing at a rapid pace. Due to this rapid advancement in technology, most manual tasks are now performed automatically, for example by robots, and therefore, robot manufacturing has grown tremendously. Various parameters require calibration during the use of a robot, and one of these parameters is the kinematic parameters. The kinematic parameters of a robot describe the geometry of the robot's motion and are used to relate the position of the robot's end-effector to the position of the robot's base via joint values. If the kinematic parameters are not calibrated, they will have large errors due to robot manufacturing defects, continuous wear during use, and accumulated zero-position offsets of the joints. As a result, an accurate calibration method is desired for robots. A widely used set of kinematic parameters are the Denavit-Hartenberg (DH) parameters, which allow the position of the robot's end-effector to be calculated based on the joint values. Other sets of kinematic parameters include the transitions between the camera, the robot base coordinate system, and the robot's gear reduction ratio.

Traditionally, calibration has been performed using laser tracker systems, where one part of the laser tracker system measures the robot's body position and another part of the laser tracker system measures the robot's end-effector position. However, such calibration is complex and inefficient due to its manual nature, and unnecessary equipment is costly. Another approach involves the introduction of cameras onto the robot, which are used at multiple positions on the robot. Furthermore, markers are placed close enough to the camera's beam to calculate accurate vectors for each robot position. However, all robot positions used for calibration must be close enough to the camera's beam. This means that motion parameters can only be calibrated within the robot's limited configuration space, which is a technical challenge.

In light of the above discussion, it is desirable to address the above shortcomings associated with robot calibration.

The present disclosure relates to a robotic system having a robot, and a method for calibrating the robot. It is an object of the present disclosure to at least partially overcome problems encountered in the prior art by providing an improved robotic system having a robot, and an improved method for automatically calibrating the robot's motion parameters, for example, to reduce errors in the position and orientation of the robot's end effector and to increase the accuracy of the robot's manipulation execution.

One or more objects of the present disclosure are achieved by the accompanying independent claims. Optional or alternative implementations of the present disclosure are further defined in the dependent claims.

In one aspect, the present disclosure provides a robotic system. The robotic system includes a robot further including a movable part, the movable part having at least one calibration marker provided thereon. The robotic system further includes at least a first joint and a second joint disposed at one end of at least one link, the first joint and the second joint configured to generate movement in the movable part controlled by a computing unit. The robotic system further includes a camera configured to capture images of the at least one calibration marker at multiple positions. The robotic system further includes a computing unit including a robot control layer, an image processing layer, an axis and reduction ratio estimation layer, and a motion parameter update layer. The robot control layer is configured to control rotation of the first joint and the second joint based on a kinematic model of the robot. The robot control layer is further configured to control rotation of the first joint and/or the second joint to the position of the at least one calibration marker at multiple positions. The image processing layer is configured to process images received from the camera to determine, for each image, a marker position of the at least one calibration marker in three-dimensional space, to be output as marker position data. The axis and reduction ratio estimation layer is configured to establish a first error function based on the marker position data and determine a first rotation axis of the first joint based on the first error function. The motion parameter update layer is configured to establish a second error function based on at least the first rotation axis and determine a set of motion parameters for the motion model based on the second error function.

The robot system includes a moving part of the robot. The moving part of the robot is provided with at least one calibration marker. The calibration marker serves to analyze the position of the moving part of the robot, and a camera is used to capture multiple positions of the calibration marker. Furthermore, a movement of the moving part is generated by a first joint and a second joint, one joint value being different from a zero position, and the other joint value being at the zero position during the entire collection procedure. Furthermore, the calculation of the rotation axis parameters and the calculation of the motion parameters include minimizing an error function. Furthermore, the first joint and the second joint are controlled by a computing unit. The computing unit includes a robot control layer that controls the rotation of the first joint and/or the second joint to change the position of one of the calibration markers. The computing unit further includes an image processing layer that serves to provide marker position data by analyzing images captured by the camera. The computing unit further includes an axis and reduction ratio estimation layer that establishes a first error function and analyzes the marker position data to determine a first rotation axis of the first joint based on the first error function. The computation unit further includes a motion parameter update layer configured to analyze the first rotation axis to establish a second error function and determine a set of motion parameters for the motion model based on the second error function. Advantageously, the robot system calibrates the robot's motion parameters, including robot link lengths, joint positions and orientations, robot joint zero positions, and joint-by-joint reduction ratios. Using the calibrated parameters, the robot can achieve better accuracy in the position and orientation of the end effector during operation execution.

In an embodiment, the movable part further includes a first servo motor coupled to the first joint via a first gear system, a second servo motor coupled to the second joint via a second gear system, a first encoder provided on the first servo motor, and a second encoder provided on the second servo motor, the first encoder and the second encoder configured to collect servo values of the first servo motor and the second servo motor, respectively. Furthermore, the computing unit includes a clock synchronization layer configured to synchronize the camera with the first encoder and the second encoder, and optionally with the rotation angle provided by the encoder, so as to timestamp images captured by the camera.

Servomotors can rotate in both clockwise and counterclockwise directions. Additionally, gear systems help transmit the servomotor's movement to the joints.

Another aspect is a method for calibrating a robot including a movable part, the movable part having at least one calibration marker disposed thereon. Furthermore, at least a first joint and a second joint are disposed at one end of at least one link, the first joint and the second joint configured to generate motion in the movable part controlled by a computing unit. The method further includes a data collection procedure including positioning the at least one calibration marker at multiple positions by rotating the first joint and/or the second joint. The method further includes acquiring multiple images of the at least one calibration marker, where the multiple images are captured by a camera while the at least one calibration marker is in multiple positions. A data processing procedure is further disclosed for processing the multiple images to determine, for each image of the multiple images, a marker position (and optionally, an orientation) of the at least one calibration marker in three-dimensional space, which is output as marker position data. The method further discloses establishing a first error function based on the marker position data. Thereafter, determining a first axis of rotation of the first joint and a second axis of rotation of the second joint based on the first error function. Further, a second error function is established based on at least the first rotation axis and the second rotation axis. Further, a set of motion parameters for a motion model of the robot is determined based on the second error function.

The method achieves at least some of the advantages and technical effects of the robotic system of the present disclosure.

It should be noted that all devices, elements, circuits, units, and means described herein may be implemented with software or hardware elements, or any combination thereof. The steps performed by various entities described herein, and the functions described as being performed by various entities, are intended to mean that each entity is adapted to or configured to perform the respective step or function. Even if, in the following description of specific embodiments, specific functions or steps performed by external entities are not reflected in the description of specific detailed elements of the entities that perform the specific steps or functions, it will be apparent to those skilled in the art that these methods and functions may be implemented with each software or hardware element, or any combination thereof. It will be understood that features of the present disclosure may be combined in various combinations without departing from the scope of the present application, as defined by the appended claims.

Further aspects, advantages, features, and objects of the present disclosure will become apparent from the following detailed description of exemplary implementations and drawings, taken in conjunction with the accompanying claims.

The foregoing summary and the following detailed description of exemplary embodiments will be better understood when read in conjunction with the accompanying drawings. For the purpose of illustrating the disclosure, exemplary structures of the disclosure are shown in the drawings. However, the disclosure is not limited to the specific methods and instrumentalities disclosed therein. Moreover, those skilled in the art will appreciate that the drawings are not to scale. Wherever possible, like elements are designated by like numerals.

Embodiments of the present disclosure will now be described, by way of example only, with reference to the following drawings:

FIG. 1 is a schematic diagram of the architecture of a robotic system according to various embodiments of the present disclosure. FIG. 1 is a schematic diagram of the architecture of a robotic system according to various embodiments of the present disclosure. FIG. 1 is a block diagram of a robotic system according to an embodiment of the present disclosure. 1 is a flowchart of a method for calibrating a robot according to an embodiment of the present disclosure.

In the accompanying drawings, underlined numbers are used to identify the item the underlined number is above or next to. Non-underlined numbers refer to the item identified by the line connecting the non-underlined number to the item. When a number is not underlined and has an associated arrow, the non-underlined number is used to identify the general item to which the arrow is pointing.

The following detailed description describes embodiments of the present disclosure and how they may be practiced. Although several modes of carrying out the present disclosure are disclosed, those skilled in the art will recognize that other embodiments of carrying out or practicing the present disclosure are possible.

1A and 1B are schematic diagrams of the architecture of a robot of a robotic system according to various embodiments of the present disclosure. 1A and 1B are collectively described, with 1B being an embodiment of 1A. Referring to 1A, robots such as robot 100A (of 1A) and robot 100B (of 1B) are shown. Robot 100A includes a base 102, a movable part 104, a calibration marker 106A, a computing unit 108, a camera 110, a first servo motor 112A and a second servo motor 112B, and a first encoder 114A and a second encoder 114B. Robot 100B includes all the same elements as robot 100A (computing unit 108 not shown), and additionally, robot 100B further includes calibration markers 106B and 106C.

The disclosed robots 100A and 100B correspond to basic robots used to automatically calibrate the motion parameters of a robotic system by using an industrial camera, such as camera 110, and at least one reference marker, such as calibration marker 106A. Robot 100B further includes calibration markers 106B and 106C.

The base 102 of the robot 100A is used to provide balance to the moving part 104 of the robot 100A. The moving part 104 is the arm of the robot 100A.

Calibration markers 106A, 106B, and 106C are used to provide marker position data. Calibration markers 106A, 106B, and 106C are attached to the end effector of robot 100A. At least one calibration marker is used for calibration purposes.

Computing unit 108 may include suitable logic, circuitry, interfaces, and/or code configured to control robot 100A. Example implementations of computing unit 108 may include, but are not limited to, a central data processing device, a microprocessor, a microcontroller, a multiple instruction set computing (CISC) processor, an application specific integrated circuit (ASIC) processor, a reduced instruction set (RISC) processor, a very long instruction word (VLIW) processor, a state machine, and other processors or control circuits.

Camera 110 may include appropriate logic, circuitry, interfaces, and/or code configured to capture an image of the position of calibration marker 106A.

The first and second servo motors 112A, 112B are used to provide movement to the arms of the robot 100A . The first and second servo motors 112A, 112B are capable of moving in clockwise and counterclockwise directions. Additionally, the first and second encoders 114A, 114B are used to determine the rotation of the first and second servo motors 112A, 112B, respectively (e.g., by calculating the angle of rotation).

In one aspect, a robotic system is provided. The robotic system includes a robot 100A further including a movable part 104. The movable part 104 has at least one calibration marker provided thereon. The robotic system further includes at least a first joint and a second joint disposed at one end of at least one link, the first joint and the second joint configured to generate movement in the movable part 104 controlled by a computing unit 108. In other words, the movable part 104 corresponds to an arm of the robot 100A. The movable part 104 includes any number of joints disposed at one end of the at least one link. The movement of the at least one link is performed by rotating at least one pre-joint. Furthermore, the computing unit 108 controls at least the first joint and the second joint to cause movement in the movable part 104, which is further beneficial for the movement of at least one calibration marker, such as the movement of the calibration marker 106A. In an implementation, the computing unit 108 includes a robot control layer that stores a set of predetermined commands for the movement of the first joint and the second joint. It should be noted that the robot control layer and the computing unit 108 can be the same device or separate devices in a distributed system. Furthermore, the robot control layer sends a set of predetermined commands to the first servo motor 112A and the second servo motor 112B to cause movement therein.

According to the embodiment, the first servo motor 112A is coupled to the first joint via a first gear system, and the second servo motor 112B is coupled to the second joint via a second gear system. Furthermore, a first encoder 114A is provided on the first servo motor 112A, and a second encoder 114B is provided on the second servo motor 112B. Furthermore, the first encoder 114A and the second encoder 114B are configured to collect servo values (e.g., angles) of the first servo motor 112A and the second servo motor 112B, respectively. Furthermore, the calculation unit 108 further includes a clock synchronization layer configured to synchronize the camera 110 with the first encoder 114A and the second encoder 114B so as to timestamp the images and servo values captured by the camera 110. In other words, at least the first joint and the second joint are connected to the first servo motor 112A and the second servo motor 112B, respectively, via a gear system having a reduction ratio for each joint as a system parameter. The movements of the first servo motor 112A and the second servo motor 112B are transmitted to at least the first joint and the second joint with some reduction ratio coefficient. Furthermore, a clock synchronization layer present in the calculation unit 108 performs synchronization for time stamps. Furthermore, the clock synchronization layer synchronizes the camera 110 with the first encoder 114A and the second encoder 114B.

The robotic system further includes a camera 110 configured to capture images of at least one calibration marker at multiple positions. In an implementation, the camera 110 includes a perception layer. Furthermore, synchronization between the camera 110 and the first and second encoders 114A and 114B is performed by time-stamping the images and encoder data captured by the camera 110. For example, during movement M 1 _J , the camera 110 captures multiple images of at least one calibration marker, such as calibration marker 106A. For example, the camera 110 captures multiple images I ₁ through I _T , where T is the total number of frames for movement M 1 _J. The images are then sent to the computation unit 108 for further processing. Furthermore, during movements (e.g., movements M 1 _J and M _{1 J+1} ), the first and second encoders 114A and 114B collect information from the first servo motor 112A of joint J and the second servo motor 112B of joint J+1 and send the information to the computation unit 108. Furthermore, synchronization is performed between the encoder clock and the camera clock. In the example, the angle values (e.g., values _a1 to _aT ) are available without any synchronization. In implementation, synchronization is performed so that a time tag for each image and each angle is obtained to establish the first error function.

1A further includes a robot control layer, an image processing layer, an axis and reduction ratio estimation layer, a motion parameter update layer, and a clock synchronization layer. The robot control layer is configured to control the rotation of the first joint and the second joint based on a motion model of the robot 100A, and the robot control layer is configured to control the rotation of the first joint and/or the second joint to position at least one calibration marker at a plurality of positions. In other words, the computation unit 108 includes various layers used to perform a calibration process in the robot system. Furthermore, the robot control layer provides motion to the arm of the robot 100A to change the position of at least one calibration marker, such as the calibration marker 106A. For example, several fiducial markers Δ _i (e.g., i∈[1, 2]) are statically attached to the end effector of the robot arm at arbitrary positions and orientations. Furthermore, at least one calibration marker, such as calibration marker 106A, allows for calibration of the position and orientation of robot 100A in three-dimensional (3D) space based on its images, for example, taking into account the intrinsic parameters of camera 110 and real-world marker dimensions.

The image processing layer is configured to process images received from the camera 110 to determine, for each image, the marker position and (optionally) the orientation of at least one calibration marker in three-dimensional space, which are to be output as marker position data. In other words, image processing is performed by acquiring data from the camera 110 and outputting a series of position/orientation data for each observed marker along with a corresponding timestamp. The image processing layer of the computation unit 108 is responsible for calculating the positions and orientations of the calibration markers 106A, 106B, 106C relative to the camera 110, for example, by using the image data I ₁ -I 1 _T. For example, without loss of generality, it is assumed that the calibration markers 106A, 106B, 106C, such as the reference marker Δ _i , are visible and detectable in all images I ₁ -I 1 _T. In other cases, the image processing layer selects a subset of frames I _i1 -I 1 _T in which Δ _i is detected by the camera 110. Furthermore, the intrinsic parameters of the camera 110 may be assumed to be accurate and known. In implementation, a camera calibration procedure is also performed on the intrinsic parameters of the camera 110. Then, fiducial marker detection in images I ₁ to I _T is performed. The result of the detection procedure is a sequence C ₁ ^I to C _T ⁱ , where C _t ⁱ is the set of (x, y) coordinates of the corners of marker Δ _i in frame I _t . In the example, the pre-computed intrinsic parameters of the camera 110 and the corner coordinates C _t ^I are used to calculate the 3D position P _t ⁱ of the marker center and the 3D marker orientation O _t ⁱ with respect to the camera 110 by the perspective-n-point (PnP) method. Furthermore, the representation of P _t ⁱ is performed by a three-dimensional vector, and the representation of O _t ⁱ is performed by a 3x3 orthonormal basis matrix.

Further, the axis and reduction ratio estimation layer is configured to establish a first error function based on the marker position data and determine a first rotation axis of the first joint based on the first error function. In other words, the axis and reduction ratio estimation layer is responsible for processing the marker position/orientation data and also the encoder data, and outputs an estimated rotation axis for each joint. For example, from the geometric properties of rotation, for each i, the 3D marker coordinates _P1i ^, ..., _PTi are arranged on a circle centered at _Qj , on ^a plane orthogonal to _Vj , and according to angles _rjα1 , _... , _rjαT , and the direction basis _O1i , _... , _OTi rotates around _Vj according to _{angles rjα1, ..., rjαT} ^. _{Further ,} the rotation axis vector _Vj , _the rotation axis point _Qj ^, and the rotation rate _rj are estimated. For example, an optimization method can be applied to a loss function L(V, Q, r) that reflects the magnitude to which the above geometric properties are not satisfied for a given set of 3D vectors V, 3D points Q, and a scalar r. The desired value is found by solving the optimization problem: ( _Vj , _Qj , _rj ) = argminL(V, Q, r).

Furthermore, the motion parameter update layer is configured to establish a second error function based on at least the first rotation axis, and determine a set of motion parameters for the motion model based on the second error function. In other words, the motion parameter update layer is responsible for processing the estimated rotation axis for each joint, and outputs calibrated motion parameters. For example, L represents the set of motion parameters to be calibrated. In implementation, given the estimated rotation axis A ₁ -A _j of each joint and the initial value L ₀ of the motion parameters, the motion parameter update layer calculates the actual parameter estimates.
This is the updated parameter value
The actual rotation axis corresponding to
and the estimated axes A ₁ to A _j . Two different sets of axes, that is, the estimated axes A ₁ to A j and the actual axes A 1 to A _j, are used.
The difference between
where T is a transformation operator that applies a specific rotation and translation to the axis position and orientation parameters. In the example, the transformation operator corresponds to the transformation between the camera 110 and the world coordinate system. Furthermore, the error function
teeth,
where d ₁ (A, B) is the minimum distance between any point belonging to the A axis and any point belonging to the B axis, and d ₂ (A, B) is the difference between the direction vectors of the A and B axes. Furthermore, the minimization of D and D' is performed by using an optimization algorithm, for example, gradient descent. Note that the present disclosure is not limited to a specific definition of the error functions D and D'. The updated motion parameters
and the transformation operator T completes the motion parameter calibration procedure, which is beneficial compared to conventional approaches. In the example, the transformation T _m ^b between the marker frame at the zero position and the robot base frame is obtained by the calibrated motion parameters, and together with the transformation T _m ^c between the camera frame obtained by image processing and the marker frame at the zero position, forms the transformation T _c ^b between the camera 110 and the base 102, thus completing the hand-eye calibration.

The robot system calibrates the motion parameters of the robot 100A, including robot link lengths, joint positions and orientations, robot joint zero positions, reduction ratios for each joint position and orientation, and reduction ratios for each joint. Using the calibrated parameters, the robot 100A can achieve improved accuracy in the position and orientation of the end effector during motion execution. According to an embodiment, the calibration procedure may be performed without encoder data; multiple images containing markers are sufficient to provide the motion model parameters and hand-eye calibration.

According to an embodiment, a special reference marker type, for example, a 3x3 ChArUco marker, can be attached to the end effector to automatically determine the intrinsic parameters of the camera 110 using motion information from the marker's keypoints. In an embodiment, the marker's keypoints are extracted from the image data _I1 - _It . Furthermore, by knowing ground truth information about the relative positions of the marker's keypoints, the intrinsic camera parameters of the camera 110 are obtained. Furthermore, by using the obtained camera parameters and the positions of the marker's keypoints on the image, the 3D position _Pt ⁱ of the marker center and the 3D marker orientation _Ot ⁱ with respect to the camera 110 are calculated by the PnP method.

According to another embodiment, at least one marker is attached to each link of the robot. This setup allows arbitrary movement of the robot during calibration and online motion parameter calibration. If arbitrary movement is applied, encoder values may be required for zero-position calibration. This embodiment can be applied to any robot architecture with rotary joints that allow arbitrary link connection. The difference between this embodiment and the main disclosure is that the movement of the robot arm can be arbitrary. In this embodiment, the camera 110 captures images I ₁ , ..., I _T , where T is the total number of frames. Then, collection of rotary encoder data α ₁ ^j , ..., α _T ^j for joints j = 1, ..., J is performed. Furthermore, the positions P 1 j , ..., P _{T j} ^{and orientations O 1 j} _, ^... , O _T ^j of the markers attached to links j = ₁ , ..., J are obtained. Furthermore, rotation axis and reduction ratio estimation is performed with data P ₁ ¹ , ..., P _{T 1} ; O ₁ 1 , ..., O _T ¹ ; α ₁ ¹ , ..., α _T ¹ from the first joint to determine V 1 , Q ¹ , _r ¹ of the first joint. For each t= ₁ , ..., T, a 3D affine transformation τ _t ¹ is calculated, which corresponds to a rotation in 3D space around the axis A ₁ = (V ₁ , Q ₁ ) by an angle value (-r ₁ α _t ¹ ). Furthermore, for t=1, ..., T,
Represents a new position.
corresponds to the position of the second joint marker as if the first joint were stationary and in its zero position. The rotation axis and reduction ratio estimation is then performed using the transformed data of the second joint to find V ₂ , Q ₂ , r _{2 .}
Next, τ _t ² is found, which corresponds to a rotation in 3D space around axis A ₂ =(V ₂ ,Q ₂ ) by an angle value (-r ₂ α _t ² ). Next, the position and orientation of joint 3:
The same sequence of operations is repeated for the remaining joints until the rotation axes A ₁ , ..., A _J at the zero position are obtained. Further, a tuning procedure is performed to determine the motion parameters based on the rotation axes A ₁ , ..., A _J using the second error function as described above.

In another embodiment, the camera 110 is attached to the end effector, and the reference markers are not attached to the end effector, but to a fixed workspace visible to the camera 110. In this embodiment, image processing is performed to obtain a series of marker positions P ₁ , ..., P _T and a series of marker orientation bases O ₁ , ..., O _T. Then, rotation axis and reduction ratio estimation is performed by using the position-orientation data of the camera 110, such as -P ₁ , ..., -P _T and O ₁ ^-1 , ..., O _T ^-1 , and finally, motion parameter adjustment is performed.

In other embodiments, multiple cameras are used in case a marker cannot be captured due to spatial obstructions. Camera 110 is preferably in a stationary position. In other embodiments, instead of using multiple cameras, camera 110 can have a predefined trajectory in case a spatial obstruction is present.

Figure 2 is a block diagram of a robotic system according to an embodiment of the present disclosure. Figure 2 is described with elements from Figures 1A and 1B. Referring to Figure 2, a robotic system 200 for calibrating a robot is shown. The robotic system 200 includes a robot control 204, a robot servo 206, an encoder layer 208, sensing 210, image processing 212, clock synchronization 214, axis and reduction ratio estimation 216, and motion parameter update 218. The robotic system 200 further includes a robot 100A (or 100B), a computing unit 108, and a camera 110. Figure 2 is described step by step from steps 202A to 202K.

The robot control 204 of the robot system 200 is used to control the robot 100A by providing commands for each movement. The encoder layer 208 in the robot system 200 is used to capture angles and movements. Sensing 210 may include the camera 110 for gathering visual information. Image processing 212 is a layer present in the computation unit 108 for analyzing images captured by the camera 110. Clock synchronization 214 is a layer present in the computation unit 108 for synchronizing the camera 110 with the encoder layer 208. Furthermore, axis and reduction ratio estimation 216 should provide estimated rotation axes for each joint. Furthermore, motion parameter update 218 is used to provide calibrated motion parameters.

In step 202A, the robot control 204 in the computing unit 108 provides commands for movement. The robot control 204 stores a set of predetermined commands for movement of at least the first joint and the second joint. Note that the robot control 204 and the computing unit 108 may be the same device.

In step 202B, the robot control 204 sends a set of predetermined commands to the robot servo 206 to cause movement in the robot servo 206.

In step 202C, the angle of the robot servo 206 is captured by the encoder layer 208. In other words, at least the first joint and the second joint are connected to the robot servo 206 via a gear system with a gear reduction ratio for each joint as a system parameter. The movement of the robot servo 206 is transmitted to at least the first joint and the second joint with some reduction ratio coefficient. Furthermore, the clock synchronization 214 performs synchronization for time stamps.

In step 202D, vision methods are collected by sensing 210. In step 202E, clock synchronization 214 synchronizes the camera 110 with the encoder layer 208. In other words, the perception layer 112 is within the camera 110. Synchronization between the camera 110 and the encoder layer 208 is performed by time-stamping images captured by the camera 110. For example, during movement M 1 _J , the camera captures multiple images I ₁ through I 1 _T of at least one calibration marker, where T is the total number of frames for movement M 1 _J. The images are sent to the computation unit 108 for processing. During movement M 1 _J , the encoder layer 208 collects information from the robot servo 206 of joint J and sends the information to the computation unit 108. Clock synchronization 214 synchronizes the encoder clock and the camera clock to obtain the servo angles a ₁ through a 1 _T.

In step 202F, image processing 212 is performed, which involves acquiring data from the camera 110 and outputs a sequence of position/orientation data with a corresponding timestamp for each observed marker. The image processing of the computation unit 108 involves calculating the position and orientation of the markers relative to the camera 110 using the image data I ₁ -I _T. Without loss of generality, the reference marker Δ _i is visible and detectable in all images I ₁ -I _T. In other cases, the image processing layer selects a subset of frames I _i1 -I i _T in which Δ _i is detected. The intrinsic parameters of the camera 110 may be assumed to be accurate and known. If not, a camera calibration procedure is performed. Furthermore, reference marker detection in the images I ₁ -I _T is performed. The result of the detection procedure is a sequence C ₁ ^I -C _T ⁱ , where C _t ⁱ is the set of (x,y) coordinates of the corners of marker Δ _i in frame I _t . Furthermore, the pre-computed intrinsic parameters and corner coordinates C _t ⁱ of the camera 110 are used to calculate the 3D positions P _t ⁱ and 3D marker orientations O _t ⁱ of the marker centers with respect to the camera 110 by the perspective-n-point (PnP) method. In the example, the representation of the marker centers P _t ⁱ is performed by a three-dimensional vector, and the representation of O _t ⁱ is performed by a 3×3 orthonormal basis matrix.

In step 202G, the visual information collected by sensing 210 is time-stamped by clock synchronization 214. Collectively in steps 202H and 202I, axis and reduction ratio estimation is performed after calculating the position and orientation of the calibration markers 106A, 106B, 106C relative to the camera 110. Axis and reduction ratio estimation 216 involves processing the calibration marker position/orientation data and encoder data to output estimated rotation axes for each joint.

In step 202J, the estimated rotation axes for each joint are processed by motion parameter update 218. Motion parameter update 218 is responsible for processing the estimated rotation axes for each joint and outputs calibrated motion parameters.

In step 202K, the motion parameter calibration procedure is completed. The updated motion parameters and transformation operators complete the motion parameter calibration procedure. The transformation between the marker frame at the zero position and the robot base frame obtained using the calibrated motion parameters, together with the transformation between the camera frame obtained by image processing and the marker frame at the zero position, forms the transformation between the camera 110 and the base 102, thus completing the hand-eye calibration.

The robot system 200 calibrates the motion parameters of the robot 100, including robot link lengths, joint positions and orientations, robot joint zero positions, reduction ratios for each joint position and orientation, and robot joint zero positions. Using the calibrated parameters, the robot system 200 can achieve improved accuracy in the position and orientation of the end effector during motion execution.

Figure 3 is a flowchart of a method for calibrating a robot according to an embodiment of the present disclosure. Figure 3 is described in conjunction with elements from Figures 1A, 1B, and 2. Figure 3 is explained step by step from steps 302 to 304.

The method 300 is used to calibrate a robot 100A having a movable part 104, the movable part 104 having at least one calibration marker thereon, and at least a first joint and a second joint disposed at one end of at least one link, the first joint and the second joint configured to generate a movement in the movable part 104 controlled by a computing unit 108. In other words, the movable part 104 forms an arm of the robot 100A. The movable part 104 may have any number of joints. The joints may be disposed at either end of at least one link. The movement of at least one link is performed by rotating an adjacent joint. The computing unit 108 controls the first joint and the second joint to impart movement to the movable part 104, and data is collected by evaluating the movement of the at least one calibration marker. For example, in FIG. 3, the method 300 begins at step 302 by performing a data collection procedure. The data collection procedure 302 includes, in step 302A, positioning at least one calibration marker at multiple positions by rotating the first joint and/or the second joint. In other words, data is collected by changing the position of at least one calibration marker, such as calibration marker 106A. Furthermore, the first joint and/or the second joint may be connected to the first servo motor 112A and the second servo motor 112B via a gear system with system parameters including a gear reduction ratio for each joint. The movement of the first servo motor 112A and the second servo motor 112B is imparted to the first joint and/or the second joint, which in turn imparts the movement to the at least one calibration marker.

The data collection procedure 302 further includes acquiring multiple images of the at least one calibration marker at step 302B. In implementation, the multiple images are captured by the camera 110 when the at least one calibration marker is in multiple positions. In other words, synchronization between the camera 110 and the first and second encoders 114A, 114B is performed by time-stamping the images captured by the camera 110. For example, during movement M 1 _J , the camera 110 captures multiple images I ₁ through I 1 _T of the at least one calibration marker, where T is the total number of frames for movement M 1 _J. The images are sent to the computation unit 108 for processing. During movement M 1 _J , the first and second encoders 114A, 114B collect information from the first and second servo motors 112A, 112B of joint J and send the information to the computation unit 108.

The method 300 further comprises performing a data processing procedure at step 304. In other words, the collected data is further processed by evaluating the movement of at least one calibration marker. The procedure 304 is further divided into various steps, such as steps 304A, 304B, 304C, 304D, and 304E.

The data processing procedure 304 includes, at 304A, processing the plurality of images to determine, for each image of the plurality of images, a marker position of at least one calibration marker in three-dimensional space, to be output as marker position data. In other words, the image processing is performed by acquiring data from the camera 110 and outputting a series of position/orientation data with a corresponding timestamp for each observed marker. In an implementation, the image processing layer of the computation unit 108 is responsible for calculating the position and orientation of the marker relative to the camera 110, for example, by using the image data I ₁ -I 1 _T. In an implementation, without loss of generality, the reference marker Δ _i is visible and detectable in all images I ₁ -I 1 _T. In another implementation, the image processing layer selects a subset of frames I ₁₁ -I 1 _T in which Δ _i is detected. The intrinsic parameters of the camera 110 may be assumed to be accurate and known. If not, a camera calibration procedure is performed. Furthermore, reference marker detection in the images I ₁ -I 1 _T is performed. The result of the detection procedure is a sequence C ₁ ^I to C _T ⁱ , where C _t ⁱ is the set of (x,y) coordinates of the corners of marker Δ _i at frame I _t . Furthermore, the pre-computed intrinsic parameters of the camera and the corner coordinates C _t ^I are used to calculate the 3D positions P _t ⁱ and 3D marker orientations O _t ⁱ of the marker centers with respect to the camera by the perspective-n-point (PnP) method. In the example, the representation of marker center P _t ⁱ is performed by a 3D vector, and the representation of O _t ⁱ is performed by a 3x3 orthonormal basis matrix.

The data processing procedure 304 further includes, in step 304B, establishing a first error function based on the marker position data. In other words, the marker position data is essentially data obtained by image processing. During image processing, the positions of at least one calibration marker in the image captured by the camera 110 are marked. These marked positions of the at least one calibration marker are referred to as marker position data. Furthermore, the axis and reduction ratio estimation layer is responsible for processing the marker position/orientation data and encoder data, and outputs estimated rotation axes for each joint. For example, from the geometric properties of rotation, for each i, the 3D marker coordinates _P1i ^, ..., _PTi ^are arranged on a circle centered at _Qj , on _{a plane orthogonal to Vj, and according to angles rjα1, ..., rjαT , and the} directional basis _O1i , ^{..., OTi} _rotates around _Vj according to angles ^rjα1 , _... , _rjαT . Furthermore, the rotation axis vector _Vj , the rotation axis point _{Qj ,} and the rotation rate _rj are estimated. For example, an optimization method can be _applied to a loss function L(V, Q, r) that reflects the extent to which _the above geometric properties are not satisfied for a given set of 3D vector V, 3D point Q, and scalar r. The desired value is found by solving the optimization problem: ( _Vj , _Qj , _rj ) = argminL(V, Q, r).

The data processing procedure 304 further includes, in step 304C, determining a first rotation axis for the first joint and a second rotation axis for the second joint based on the first error function. In other words, based on the calculated marker position/orientation data and the servo motor rotation data collected by the first encoder 114A and the second encoder 114B, the information is processed to determine the rotation axis for each joint and the reduction ratio after the joint.

The data processing procedure 304 further includes, in step 304D, establishing a second error function based on at least the first and second rotational axes. After determining the first and second rotational axes of the first and second joints based on the first error function, a second error function is established using the first and second rotational axes. In particular, one or more error functions may be established based on the calculated 3D positions of the marker. Furthermore, parameters are calculated to define a set of rotational axes of the robot system 200 based on the error function.

The data processing procedure 304 further includes, at step 304F, determining a set of motion parameters for the motion model of the robot 100A based on the second error function. In other words, after determining the second error function, the set of motion parameters is determined using the second error function. In particular, the motion parameter update layer is responsible for processing the estimated rotation axis for each joint and outputs calibrated motion parameters.

According to an embodiment, the method 300 includes calibrating a kinematic model of the robot 100A using the determined set of kinematic parameters. In other words, the set of kinematic parameters is used to calibrate the kinematic model of the robot 100A. Further calibration is performed, including the robot link lengths, joint positions and orientations, robot joint zero positions, and reduction ratios for each joint.

According to the embodiment, the movable part 104 further includes a first servo motor 112A configured to move the first joint, a second servo motor 112B configured to move the second joint, a first encoder 114A provided on the first servo motor 112A, and a second encoder 114B provided on the second servo motor 112B. Furthermore, the camera 110 is synchronized with the first encoder 114A and the second encoder 114B, and the data collection procedure further includes time-stamping each image of the plurality of images. Furthermore, the time-stamp corresponds to a time step within the series of time steps. In other words, a continuous circular movement of each joint is performed. During the joint movement, motor rotary encoder data is collected, and images of markers are placed on the end effector. Further, image processing and calculation of the marker position and orientation are performed. Furthermore, the calculated marker position/orientation data is based on the motor rotation data collected by the first encoder 114A and the second encoder 114B. The information is then processed to determine the rotation axis for each joint and the reduction ratio for each joint.

According to an embodiment, the first encoder 114A and the second encoder 114B are configured to collect servo values of the first servo motor 112A and the second servo motor 112B, respectively. The data processing procedure further includes determining, for each of the first servo motor 112A and the second servo motor 112B, a plurality of servo motor angles for each servo motor based on the servo values collected by the first encoder 114A and the second encoder 114B, respectively. In other words, at least the first joint and the second joint are connected to the first servo motor 112A and the second servo motor 112B via a gear system having a gear reduction ratio for each joint as a system parameter. The movement of the first servo motor 112A and the second servo motor 112B is transmitted to at least the first joint and the second joint with some reduction ratio coefficient.

According to an embodiment, the marker positions are determined for each image of the plurality of images by extracting from the image a plurality of corner coordinates of at least one calibration marker. Further, each of the plurality of corner coordinates corresponds to a position that is approximately at a corner of the at least one calibration marker. In other words, the image processing layer of the computation unit 108 is responsible for calculating the position and orientation of the calibration markers relative to the camera 110 using the image data I ₁ -I _T. Furthermore, without loss of generality, it is assumed that the reference marker Δ _i is visible and detectable in all images I ₁ -I _T.

According to an embodiment, the marker positions are determined for each image of the multiple images by calculating the marker center coordinates using multiple corner coordinates of at least one calibration marker on that image and one or more intrinsic camera parameters. In other words, the at least one calibration marker undergoes a circular motion while the joint rotates. Furthermore, the angular trajectory of the joint needs to be large enough for the camera 110 to capture multiple distinguishable positions of the marker Δ _i . Furthermore, for each j = 1, ..., J, the sequence of data collection, image processing, and rotation axis and reduction ratio estimation is repeated to obtain an estimate of the rotation axis position and orientation for each joint A _j defined by a tuple (V _j , Q _j ), where V _j is a 3D rotation vector defining the orientation of the j-th rotation axis, and Q _j is a 3D point defining the rotation axis position and reduction ratio r _j .

According to an embodiment, the method 300 includes processing a plurality of images, which further includes determining, for each image of the plurality of images, a marker orientation of at least one calibration marker in three-dimensional space based on a plurality of corner coordinates of the at least one calibration marker on the image and one or more intrinsic camera parameters, to output as marker position data. For example, without loss of generality, a fiducial marker Δ _i is visible and detectable in all images I ₁ to I _t . In implementation, the image processing layer selects a subset of frames I _i1 to I _t in which Δ _i is detected. The intrinsic parameters of the camera 110 may be assumed to be accurate and known. If not, a camera calibration procedure is performed. Further, fiducial marker detection in the images I ₁ to I _t is performed. The result of the detection procedure is a sequence C ₁ ^I to C _t ⁱ , where C _t ⁱ is the set of (x, y) coordinates of the corners of marker Δ _i in frame I _t . Furthermore, the pre ^- computed intrinsic parameters of the camera and the corner coordinates _CtI are used to calculate ^the 3D positions _PtI and 3D marker orientations _OtI of the marker centers with respect to the camera by the perspective- ⁿ -point (PnP ⁾ method. ^In the example, the representation of the marker center _PtI is performed by a 3D vector, and the representation of _OtI is performed by a 3x3 orthonormal basis matrix.

According to an embodiment, the method 300 comprises establishing a first error function comprising an arc-fit between a marker center coordinate and a marker orientation of at least one calibration marker corresponding to a circular movement of the at least one marker relative to the coordinate system of the camera 110. In other words, the error function is based on an arc-fit between a 3D position of the marker and a 3D marker orientation corresponding to a circular movement of the at least one calibration marker relative to another marker or the camera coordinate system, and optionally also based on encoder data and a fitting of the encoder data to the 3D orientation of the at least one calibration marker.

According to the embodiment, the movable part 104 includes a first servo motor 112A coupled to a first joint via a first gear system having a gear reduction ratio. The method 300 further includes determining a first rotation axis of the first joint, which includes establishing a first error function based on marker center coordinates and marker orientations of at least one calibration marker determined from the multiple images, and applying an optimization function to the first error function to determine a joint rotation axis origin Q, a rotation axis vector V, and a gear reduction ratio r. In other words, the movable part 104 includes at least two joints. The joints are located at one end of at least one link, and at least one link movement is achieved by rotation of at least one pre-joint. Furthermore, at least the first joint and the second joint are connected to the first servo motor 112A and the second servo motor 112B via a gear system having a gear reduction ratio for each joint as a system parameter. The motion of the first servo motor 112A and the second servo motor 112B is transmitted to at least the first joint and the second joint with some reduction ratio coefficient. Furthermore, the axis and reduction ratio estimation layer is responsible for processing the marker position/orientation data and encoder data, and outputs an estimated rotation axis for each joint. Furthermore, the motion parameter update layer is responsible for processing the estimated rotation axis for each joint, and outputs calibrated motion parameters.

According to an embodiment, a first encoder 114A is provided to the first servo motor 112A, configured to output encoder data including a servo value of the first servo motor 112A, and a gear reduction ratio is determined based on the marker orientation of at least one calibration marker determined from multiple images and the encoder data of the first servo motor 112A. In other words, a gear system having a gear reduction ratio for each joint as a system parameter is used to connect at least the first joint and the second joint with the first servo motor 112A and the second servo motor 112B. At least the first joint and the second joint are moved by the movement of the first servo motor 112A and the second servo motor 112B. Furthermore, any reduction ratio coefficient is also applied along with the movement.

According to an embodiment, at least one calibration marker includes one or more marker keypoints. Further, the marker keypoints correspond to uniquely identifiable points on the at least one calibration marker, and the data processing procedure further includes extracting the one or more marker keypoints from the multiple images. For example, a special fiducial marker type, e.g., a 3x3 ChArUco marker, may be attached to the end effector to automatically determine the intrinsic parameters of the camera 110 using motion information from the marker keypoints.

According to an embodiment, the method includes determining one or more intrinsic camera parameters based on the relative positions of one or more marker keypoints. For example, by extracting the marker keypoints from the image data I ₁ , ..., I _T. Furthermore, by knowing ground truth information about the relative positions of the marker keypoints, the intrinsic camera parameters of the camera 110 need to be obtained. Furthermore, by using the obtained camera parameters and the positions of the marker keypoints on the image, the 3D position P _t ⁱ of the marker center and the 3D marker orientation O _t ⁱ are calculated with a PnP method.

According to an embodiment, the movable part 104 includes a plurality of J joints arranged to be rotated by a corresponding plurality of J servo motors. Furthermore, positioning the at least one calibration marker includes controlling the plurality of J servo motors to execute a set of motions _M1 , _M2 , ..., MJ to rotate the plurality of _J joints. In other words, a set of predefined commands stored in the computing unit 108 is sent to the first servo motor 112A and the second servo motor 112B. The first servo motor 112A and the second servo motor 112B execute the set of motions _M1 , ..., _MJ , where J is the number of joints in the movable part 104 and _Mj is determined by moving the jth robot joint while the other joints remain stationary at their zero positions. The joints move, and at least one calibration marker, such as calibration marker 106A, is statically attached to the movable part of the robot at an arbitrary position and orientation. As a result, the at least one calibration marker performs a circular motion while the joints rotate. The angular trajectory of the joints needs to be large enough for the camera 110 to capture multiple distinct positions of the marker Δ _i . Furthermore, for each j = 1,...,J, the sequence of data collection, image processing, and rotation axis and reduction ratio estimation is repeated to obtain an estimate of the rotation axis position and orientation for each joint A _j , defined by the tuple (V _j ,Q _j ). Furthermore, V _j is a 3D rotation vector defining the orientation of the j-th rotation axis, and Q _j is a 3D point defining the rotation axis position and reduction ratio r _j .

According to an embodiment, the movement _Mn of the nth joint, where 1≦n≦J, is determined by controlling the nth servo motor to rotate the nth joint while keeping the remaining joints of the plurality of J joints at their zero positions. In other words, the number of joints is equal to the number of servo motors, and as the servo motors move, movement is imparted to the joints. This movement is controlled by the calculation unit 108. If only one servo motor is rotating, then only one joint is moving, and in this case, the remaining joints are considered to be at their zero positions.

According to an embodiment, at least one calibration marker is provided on one or more links each connecting two adjacent joints of the plurality of J joints, and an encoder is provided on each servo motor of the plurality of J servo motors, each of the plurality of encoders configured to output encoder data corresponding to each joint of the plurality of J joints. Further, the encoder data includes a servo value (e.g., an angle) of the corresponding servo motor. The movement of the servo motor translates into a rotation of the corresponding joint. When representing the movement, the encoder generates encoder data, where the encoder estimates the angle of rotation of the joint movement, which corresponds to the servo value.

According to an embodiment, the set of motions M ₁ , M ₂ , ..., M _J is arbitrary, and processing the plurality of images includes, for each n-th joint of the plurality of J joints, for each image of the plurality of images, transforming a position of at least one calibration marker between the (n-1)-th joint and the n-th joint from the motion frame of the (n-1)-th joint to a stationary frame of the (n-1)-th joint in which the (n-1)-th joint is stationary and at a zero position, using the encoder data of the (n-1)-th joint.

According to an embodiment, the method 300 includes determining a set of motion parameters, where determining the set of motion parameters includes determining a link length between a first joint and a second joint. In other words, the link length is determined between the first joint and the second joint by a process executed by the computation unit 108. Furthermore, motion parameters associated with all links and joints located between the robot base and the position of the at least one calibration marker are calibrated, except for the first and last link lengths, respectively.

According to an embodiment, determining the set of motion parameters includes: determining a marker-based transformation between the frame of at least one calibration marker at the zero position and the frame of the base 102 of the robot 110A using the set of motion parameters; determining a marker-camera transformation between the frame of the at least one calibration marker at the zero position and the frame of the camera 110 by processing a plurality of images; and calculating a camera-based transformation between the frame of the camera 110 and the frame of the base of the robot based on the determined marker-based transformation and the determined marker-camera transformation. In other words, the motion parameter update layer is responsible for processing the estimated rotation axis for each joint and outputs calibrated motion parameters. For example, the set of motion parameters to be calibrated is denoted as L. Furthermore, given the estimated rotation axis A ₁ to A _j of each joint and the initial value of the motion parameters L ₀ , the motion parameter update layer calculates the actual parameter estimates.
This is the updated parameter value
The actual rotation axis corresponding to
and the estimated axes A ₁ to A _j . Two different sets of axes, that is, the estimated rotation axes A ₁ to A j and the actual rotation axes A 1 to A _j, are used.
The difference between
where T is a transformation operator that applies a particular rotation and translation to the axis position and orientation parameters.

Method 300 calibrates robot motion parameters, including robot link lengths, joint positions and orientations, robot joint zero positions, reduction ratios for each joint position and orientation, robot joint zero positions, and reduction ratios for each joint. Using the calibrated parameters, method 300 can achieve improved accuracy of the end effector position and orientation during motion execution.

Steps 302 and 304 are merely exemplary, and other alternatives may be provided, in which one or more steps are added, one or more steps are removed, or one or more steps are provided in a different order, without departing from the scope of the claims herein.

A non-transitory computer-readable storage medium is further provided having machine-readable code that, when executed by a processor, causes the processor to position at least one calibration marker at a plurality of positions by rotating a first joint and/or a second joint. The processor further acquires a plurality of images of the at least one calibration marker, the plurality of images being captured by the camera 110 when the at least one calibration marker is in the plurality of positions. The processor further processes the plurality of images to determine, for each image of the plurality of images, a marker position (and optionally, an orientation) of the at least one calibration marker in three-dimensional space to be output as marker position data. The processor further establishes a first error function based on the marker position data. The processor further determines a first rotation axis of the first joint and a second rotation axis of the second joint based on the first error function. The processor further establishes a second error function based on at least the first rotation axis and the second rotation axis. The processor further determines a set of motion parameters for a motion model of the robot 110A based on the second error function. In examples, the instructions are embodied in a computer-readable medium, including, but not limited to, an electrically erasable programmable read-only memory (EEPROM), a random access memory (RAM), a read-only memory (ROM), a hard disk drive (HDD), flash memory, a secure digital (SD) card, a solid-state drive (SSD), a computer-readable storage medium, and/or a CPU cache memory. In examples, the instructions are generated by a computer program, and the computer program is executed in view of method 300 for use in implementing method 300 by a processor.

Changes to the embodiments of the present disclosure described above are possible without departing from the scope of the present disclosure, as defined by the appended claims. The terms "comprising," "having," "incorporating," "having," "being," and the like, used to describe and claim the present disclosure, are intended to interpret the disclosure in a non-exhaustive manner, meaning that there may be items, components, or elements not expressly described. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "provided in some embodiments but not in other embodiments." While certain features of the present disclosure are described in the context of separate embodiments for clarity, it will be understood that they may also be provided in combination in a single embodiment. Conversely, various features of the invention, while described in the context of a single embodiment for brevity, may also be provided separately or in any suitable combination or with any other described embodiment of the present disclosure, as appropriate.

Claims (20)

a robot having a movable part, the movable part having at least one calibration marker thereon, and at least a first joint and a second joint disposed at one end of at least one link, the first joint and the second joint configured to generate a movement in the movable part controlled by a computing unit;
a camera configured to capture a plurality of images of the at least one calibration marker at a plurality of positions;
the computation unit including a robot control layer, an image processing layer, an axis and reduction ratio estimation layer, and a motion parameter update layer;
the robot control layer is configured to control rotations of the first joint and the second joint based on a kinematic model of the robot, and the robot control layer is configured to control rotations of the first joint and/or the second joint to position the at least one calibration marker at the plurality of positions;
the image processing layer is configured to process the plurality of images received from the camera to determine, for each image of the plurality of images, a marker position of the at least one calibration marker in three-dimensional space to be output as marker position data;
the axis and reduction ratio estimation layer is configured to establish a first error function based on the marker position data and determine a first axis of rotation of the first joint based on the first error function;
the motion parameter update layer is configured to establish a second error function based on at least the first rotation axis, and determine a set of motion parameters for the motion model based on the second error function;
the image processing layer is further configured to, for each image of the plurality of images, determine the marker position by extracting from the image a plurality of corner coordinates of the at least one calibration marker, each of the plurality of corner coordinates corresponding to a position that is approximately at a corner of the at least one calibration marker.
Robot system. the movable part further comprises a first servo motor coupled to the first joint via a first gear system, a second servo motor coupled to the second joint via a second gear system, a first encoder provided on the first servo motor, and a second encoder provided on the second servo motor, the first encoder and the second encoder being configured to collect servo values of the first servo motor and the second servo motor, respectively; and the calculation unit further comprises a clock synchronization layer configured to synchronize the camera with the first encoder and the second encoder so as to timestamp each image of the plurality of images captured by the camera.
The robot system of claim 1 . 1. A method for calibrating a robot having a movable part, the movable part having at least one calibration marker provided thereon, and at least a first joint and a second joint arranged at one end of at least one link, the first joint and the second joint being configured to generate a movement in the movable part controlled by a computing unit, the method comprising:
In the data collection procedure,
positioning the at least one calibration marker at a plurality of positions by rotating the first joint and/or the second joint;
acquiring a plurality of images of the at least one calibration marker, the plurality of images being captured by a camera when the at least one calibration marker is in the plurality of positions;
In the data processing procedure,
processing the plurality of images to determine, for each image of the plurality of images, a marker position of the at least one calibration marker in three-dimensional space to be output as marker position data;
establishing a first error function based on the marker position data;
determining a first axis of rotation of the first joint and a second axis of rotation of the second joint based on the first error function;
establishing a second error function based on at least the first axis of rotation and the second axis of rotation;
determining a set of motion parameters for a motion model of the robot based on the second error function;
the marker positions are determined by, for each image of the plurality of images, extracting from the image a plurality of corner coordinates of the at least one calibration marker, each of the plurality of corner coordinates corresponding to a position approximately at a corner of the at least one calibration marker.
method. calibrating the kinematic model of the robot using the determined set of kinematic parameters.
The method of claim 3. the movable part further comprises a first servo motor configured to move the first joint, a second servo motor configured to move the second joint, a first encoder provided on the first servo motor, and a second encoder provided on the second servo motor, the camera is synchronized with the first encoder and the second encoder, and the data collection procedure further comprises assigning a time stamp to each image of the plurality of images, the time stamp corresponding to a time step within a series of time steps.
The method according to claim 3 or 4. the first encoder and the second encoder are configured to collect servo values of the first servo motor and the second servo motor, respectively, and the data processing procedure further comprises, for each of the first servo motor and the second servo motor, determining a plurality of servo motor angles for each servo motor based on the servo values collected by the first encoder and the second encoder, respectively.
The method of claim 5. the marker positions are determined by calculating, for each image of the plurality of images, a marker center coordinate using the plurality of corner coordinates of the at least one calibration marker on that image and one or more intrinsic camera parameters.
The method of claim 5. and processing the plurality of images further comprises determining, for each image of the plurality of images, a marker orientation of the at least one calibration marker in three-dimensional space based on the plurality of corner coordinates of the at least one calibration marker on that image and one or more intrinsic camera parameters, to output as the marker position data.
The method of claim 7. establishing the first error function includes arc-fitting the marker center coordinates and the marker orientation of the at least one calibration marker corresponding to a circular motion of the at least one calibration marker relative to a coordinate system of the camera.
The method of claim 8. the movable part includes a first servo motor coupled to the first joint via a first gear system having a gear reduction ratio, and determining the first rotation axis of the first joint includes establishing the first error function based on the marker center coordinates and the marker orientation of the at least one calibration marker determined from the plurality of images, and applying an optimization function to the first error function to determine a joint rotation axis origin, a rotation axis vector, and the gear reduction ratio .
10. The method according to claim 8 or 9. a first encoder is provided on the first servo motor configured to output encoder data including a servo value of the first servo motor, and the gear reduction ratio is determined based on the marker orientation of the at least one calibration marker determined from the plurality of images and the encoder data of the first servo motor.
The method of claim 10. the at least one calibration marker includes one or more marker keypoints, the marker keypoints corresponding to uniquely identifiable points on the at least one calibration marker, and the data processing procedure comprises:
extracting the one or more marker keypoints from the plurality of images;
determining one or more intrinsic camera parameters based on the relative positions of the one or more marker keypoints.
12. The method according to any one of claims 3 to 11. the movable part has a plurality of J joints arranged to be rotated by a corresponding plurality of J servo motors, and positioning the at least one calibration marker comprises controlling the plurality of J servo motors to perform a set of movements M ₁ , M ₂ , ..., M _J to rotate the plurality of J joints.
13. The method according to any one of claims 3 to 12. where 1≦n≦J, the movement _Mn of an n-th joint is determined by controlling an n-th servo motor to rotate the n-th joint while keeping the remaining joints of the plurality of J joints at their respective zero positions.
The method of claim 13. at least one calibration marker is provided on one or more links connecting two adjacent joints of the plurality of J joints, and a plurality of encoders are provided on corresponding servo motors of the plurality of J servo motors, each of the plurality of encoders being configured to output encoder data corresponding to each joint of the plurality of J joints, the encoder data including a servo value of the corresponding servo motor;
15. The method of claim 13 or 14. The set of motions M ₁ , M ₂ , ..., M _J is arbitrary, and processing the plurality of images includes, for each n-th joint of the plurality of J joints, transforming, for each image of the plurality of images, a position of the at least one calibration marker between the (n-1)-th joint and the n-th joint from a motion frame of the (n-1)-th joint to a stationary frame of the (n-1)-th joint in which the (n-1)-th joint is stationary and at a zero position, using the encoder data of the (n-1)-th joint.
16. The method of claim 15. determining the set of motion parameters includes determining a link length between the first joint and the second joint;
17. The method according to any one of claims 3 to 16. determining the set of motion parameters includes determining a marker-based transformation between a frame of the at least one calibration marker at a zero position and a frame of a base of the robot using the set of motion parameters, determining a marker-to-camera transformation between the frame of the at least one calibration marker at the zero position and a frame of the camera by processing the plurality of images, and calculating a camera-based transformation between the frame of the camera and the frame of the base of the robot based on the determined marker-based transformation and the determined marker-to-camera transformation.
18. The method according to any one of claims 3 to 17. a processor that executes the program; and a program that executes the program. The program includes machine-readable code that, when executed by a processor, causes the processor to:
positioning at least one calibration marker provided on a movable part of a robot at a plurality of positions by rotating a first joint and/or a second joint, the movable part having at least the first joint and the second joint disposed at one end of at least one link, the first joint and the second joint configured to produce a movement in the movable part controlled by the processor;
acquiring a plurality of images of the at least one calibration marker, the plurality of images being captured by a camera when the at least one calibration marker is in the plurality of positions;
processing the plurality of images to determine, for each image of the plurality of images, a marker position of the at least one calibration marker in three-dimensional space to be output as marker position data;
establishing a first error function based on the marker position data;
determining a first axis of rotation of the first joint and a second axis of rotation of the second joint based on the first error function;
establishing a second error function based on at least the first axis of rotation and the second axis of rotation;
determining a set of motion parameters for a motion model of the robot based on the second error function;
the marker positions are determined by, for each image of the plurality of images, extracting from the image a plurality of corner coordinates of the at least one calibration marker, each of the plurality of corner coordinates corresponding to a position approximately at a corner of the at least one calibration marker.
A non-transitory computer-readable storage medium. A program that, when executed by a processor, causes the processor to perform the method of any one of claims 3 to 18.