JP3901006B2 - Visual recognition device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a visual recognition apparatus for controlling a position and orientation based on image data captured by a camera.
[0002]
[Problems to be solved by the invention]
As this type of apparatus or method, there is a so-called visual servo technique. In this case, for example, when a workpiece is gripped by a robot or the like, the position and orientation of the hand are controlled by photographing the workpiece with a camera attached to the hand.
[0003]
In this case, in the position and orientation control, a reference image is registered in advance, and the position and orientation of the hand can be quickly and accurately controlled by controlling the position and orientation so that the workpiece can be seen in the same manner as the reference image. It can be carried out. The reference image is obtained by placing the robot hand in a position where position and orientation control is desired and holding the workpiece, and photographing this state with a camera attached to the hand.
[0004]
By adopting such visual servo technology, the position and orientation of the robot hand can be accurately controlled within the resolution range of the image captured by the camera. However, even at the research level, even though it seems that no technical problem has occurred, it is clear from the inventors' consideration that the following problems occur when viewed from a practical viewpoint. It is becoming.
[0005]
As one of them, it is a problem when the object can be recognized but the movement or control based on the recognition result becomes impossible. In the position / orientation control by the visual servo, there is no limited factor for the movement route, so it is not possible to predict which route will be taken.
[0006]
Therefore, depending on the work environment, for example, as shown in FIG. 7, there may be an obstacle in the moving direction of the robot. Here, when the camera A attached to the robot moves with the workpiece W (Aa in the figure). From the position No. to the position Ab) and collide with the obstacle S. In such a case, in the control system using the visual servo, as long as the workpiece W can be recognized, there is no means for avoiding the obstacle S, and the camera A and the robot body may collide with the obstacle S.
[0007]
Such a technical problem occurs due to the work environment even though the robot is capable of following and controlling the object by visual servoing. It can be regarded as a malfunction due to a factor.
[0008]
On the other hand, the camera attached to the arm of the robot is supposed to be able to move in any direction and posture when following the target, but in reality the cable connected to the camera is rotated by the arm. If this condition is met, the cable may break. Such a technical problem is similar to a problem due to an external factor as described above, but is also a constraint condition on the robot body side, and therefore includes an internal factor.
[0009]
In addition, a robot having a multi-joint arm has a state in which, for example, two rotation axes are in a straight line. When the control state of the position and orientation is in such a state, the position and orientation of the arm are controlled. This is a so-called singular point that the solution of the control data is indefinite, and as a result, there is a possibility that control of the robot body cannot be continued. Since this technical problem is inherently a problem of the robot body, it can be considered to be caused by an internal factor.
[0010]
The present invention has been made in view of the above circumstances, and its purpose is to control the position and orientation of the control target apparatus when it interferes with controlling the position and orientation due to external factors and internal factors. Correspondingly, it is an object of the present invention to provide a visual recognition device capable of reliably performing position and orientation control regardless of these factors.
[0011]
[Means for Solving the Problems]
According to the invention of claim 1, the position and orientation of the control target device is controlled by the control means so that the image of the object photographed by the camera is in the same state as the reference image registered in advance. In this process, the movement destination based on the position / orientation control is determined based on the information of the movement prohibition area set corresponding to the position / orientation that hinders the control of the position / orientation of the control target device stored in the storage unit. When the movement-prohibited area stored in the storage means is applicable, the position and orientation are controlled outside the movement-prohibited area by avoiding this, so obstacles that hinder the movement of the control target apparatus and apparatus-specific restrictions It is possible to reliably control movement to the target position of the object without causing inconvenience due to conditions and the like.
[0012]
According to the invention of claim 2, in the above invention, since the movement prohibited area can be set as an area corresponding to the position of an obstacle existing within the movement range of the control target apparatus, If there is an area that may move according to the environment and there are obstacles, etc., this can be set as a movement prohibited area to perform flexible position and orientation control adapted to the work environment become able to.
[0013]
According to the invention of claim 3, in each of the above inventions, the movement prohibited area is configured to be set as an area corresponding to the uncontrollable condition accompanying the movement of the control target apparatus. If it is difficult to move during position and orientation control due to the characteristics of the above, if that condition is set as a movement prohibition area, this can be avoided and the position and orientation can be controlled before shifting to such a state. Therefore, the position and orientation control can be performed reliably by preventing problems such as the control target apparatus becoming uncontrollable on the way.
[0014]
According to the invention of claim 4, in the invention of claim 3, when the device to be controlled is set as an articulated robot, the movement prohibited area is set as a singular point of the arm of the articulated robot as an uncontrollable condition. Because it is a configurable configuration, it becomes possible to perform control that avoids singularities that occur in posture control due to the structure of the robot, without taking separate measures for robot-specific singularities Thus, the position and orientation can be controlled easily and reliably.
[0015]
According to the invention of claim 5, in the invention of claim 3 or 4, when the device to be controlled is set as a robot having a rotatable arm, the movement prohibited region is set as the uncontrollable condition and the robot arm This configuration prevents the rotation even if there is a risk of disconnection if the rotation amount becomes unlimited, such as when a cable is attached to the arm, for example. Position and orientation control can be reliably performed.
[0016]
According to the invention of claim 6, in each of the above inventions, since the movement prohibition area is registered as information stored and set in advance in the storage means, the position and orientation control by the control means is always stored and set in the storage means. This can be performed reliably based on the information of the movement prohibited area. Also, since information is registered as information stored and stored in advance in this way, for example, information on a movement prohibited area is created in advance as common information corresponding to the work environment, and this is introduced into the work environment. By applying the method to each control target device, it is possible to set the movement prohibited area quickly and surely and to control the position and orientation with certainty.
[0017]
According to the invention of claim 7, in each of the above inventions, a sensor for detecting a position and orientation that hinders control of the position and orientation of the device to be controlled is provided, and the control means is prohibited from moving based on the detection result of the sensor. Since the configuration is such that the area is recognized and stored in the storage means, even if information is not stored and set in advance as a movement-prohibited area, the position and orientation are actually controlled in the work environment. It is now possible to create position-and-orientation control by creating a movement-prohibited area based on the detection results, and respond quickly to changes in the work environment and changes in the situation to avoid accidents such as collisions and uncontrollable states. Thus, position and orientation control can be performed reliably.
[0018]
According to the invention of claim 8, in each of the above-mentioned inventions, the communication means capable of receiving information indicating the position and orientation that interferes with the control of the position and orientation of the device to be controlled is provided, and the control means is received by the communication means. Since the information on the movement prohibited area is recognized on the basis of the information and stored in the storage means, the movement in the position and orientation control is performed based on the information on the movement prohibited area received from the outside via the communication means. The prohibited area can be taken in reliably and quickly, and thereby the position and orientation control can be performed reliably.
[0019]
According to the invention of claim 9, in each of the above inventions, the movement prohibited area is configured to be memorable as trajectory restriction data that restricts a trajectory generated corresponding to the movement of the position and orientation of the control target device. When controlling the target device, it can be easily determined by determining whether or not the generated trajectory data corresponds to the trajectory restriction data, and if so, the trajectory restriction data should be avoided. By setting to, control can be performed easily and quickly.
[0020]
According to the invention of claim 10, in the invention of claim 9, when the control means restricts the trajectory of the control target device based on the trajectory restriction data, the gain for controlling each part is used as the trajectory restriction data. Since the trajectory is controlled by changing the setting accordingly, only gain adjustment control is performed in the same way as normal control without performing special operation in response to entering the movement prohibited area. Therefore, it is possible to perform an operation avoiding the above, and it is possible to carry out the operation easily and inexpensively.
[0021]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment in which the present invention is applied to a visual servo in an articulated robot for assembly will be described with reference to FIGS. FIG. 1 schematically shows an assembly robot system. This system includes a robot body 1 as a device to be controlled and a control device 2 that controls the robot body 1. The control device 2 includes a robot controller 3, a visual recognition device 4, and a CCD camera 5. It is configured.
[0022]
The robot body 1 is configured as a 6-axis small vertical articulated robot for assembly. In this robot body 1, a shoulder portion 7 is provided on a base portion 6 so as to be rotatable (turnable) about a vertical axis, and a pair of lower arm portions 8 extending upward at both left and right end portions of the shoulder portion 7. Are coupled to be rotatable about a horizontal axis. A front end portion of the intermediate arm portion 9 is connected to be rotatable about a horizontal axis so as to be sandwiched between upper ends of the lower arm portion 8 from both sides.
[0023]
Further, the rear end portion of the upper arm portion 10 extending forward is connected to the front end portion of the intermediate arm portion 9 so as to be coaxially rotatable, and the front end side of the upper arm portion 10 is bifurcated to the left and right. The wrist part 11 is connected so as to be rotatable about a horizontal axis. Further, a circular flange portion 12 is connected to the front end portion of the wrist portion 11 so as to be coaxially rotatable, and the flange portion 12 is provided in a state in which a hand for gripping the workpiece W as an object can be attached. ing. Although not shown, a servo motor for driving each axis is incorporated in the robot body 1.
[0024]
The CCD camera 5 described above is for photographing the workpiece W and is attached to the upper arm unit 10. A signal line for transmitting an image signal of the CCD camera 5 is provided integrally with a cable 13 for controlling the robot body 1. The robot controller 3 includes a ROM and a RAM that store a control program mainly using a CPU, and includes a servo control unit that outputs a control command to the robot body 1 as a drive signal.
[0025]
The visual recognition device 4 mainly includes a CPU for performing visual servoing, and includes a ROM and a RAM that store a program for position and orientation control, an interface circuit for inputting an image signal from the CCD camera 5, an image processing unit, The configuration includes various circuits such as an image memory.
[0026]
Then, as will be described later, the control device 2 takes in the image signal of the CCD camera 5 by the visual recognition device 4, and the control signal is in a position and orientation in which the appearance matches the reference image registered in advance. Is output to the robot controller 3, the robot controller 3 controls to drive the position and orientation of the robot body 1 to be moved based on this.
[0027]
In addition to the above configuration, it is effective to provide a communication unit capable of receiving an external signal as the configuration of the control device 2 described above. As described later, this is provided so that limit data for limiting the trajectory as the movement prohibition area can be received from the outside.
[0028]
FIG. 2 shows a functional block configuration of the control device 2. The function of the visual recognition device 4 corresponds to the visual recognition unit 14, and the functional configuration of the robot controller 3 is a model data storage unit for registering a model. 15, a trajectory generation unit 16 that generates a trajectory of the robot based on the recognized result, a restriction data storage unit 17 that is a storage means for setting and storing a movement prohibition area as referred to in the present invention, and A trajectory control unit 18 that controls the operation of the robot based on the trajectory.
[0029]
The configuration of the control device 2 is specifically composed of a microcomputer, a ROM, a RAM, and the like, and is configured to operate according to a control program described later.
[0030]
Next, the operation of this embodiment will be described with reference to FIGS.
In the position / orientation control by the visual servo, the reference image is generally registered by the following method. This is because, prior to the control, the work W is photographed by the camera 5 in a state where the target position and posture for performing the work on the work W by the robot body 1 which is a control target device is taken, and this is used as a reference image. sign up.
[0031]
Thus, the work W is photographed by the camera 5 in a state where the robot body 1 is in an arbitrary position and orientation, and the visual servo based on the image information, that is, the image of the work W photographed by the camera 5 is obtained by the control device 2. Control is performed so as to match the reference image.
[0032]
Next, the trajectory restriction data referred to in the present invention is set. This restricts the restriction data so that the robot body 1 and the camera 5 can avoid a condition in which the robot body 1 or the camera 5 collides with an obstacle in the work environment or the robot body 1 cannot be controlled. This is an operation to be stored in the data storage unit 17.
[0033]
For example, in the case shown in FIG. 4, that is, when the position and orientation are controlled so that the relative position with respect to the workpiece W becomes a predetermined target position while the workpiece W is moving, the following setting is made. To do. In the drawing, there is a possibility of collision with the obstacle S when the camera 5 moves following the moving workpiece W, and an example is shown in which this is avoided.
[0034]
Here, it is assumed that the workpiece W sequentially moves from the position of Wa to the positions of Wb and Wc along the line, and the camera 5 moves from the position of 5a to 5b and 5c while approaching the target position. Move to position. At this time, at the position 5b, the camera 5 is controlled so as to avoid the collision with the obstacle S so as to avoid the collision and to follow from a position temporarily separated. In order to perform this control, specifically, the storage unit is stored as trajectory restriction data so that it can be determined whether or not the trajectory generated as the position and orientation control data of the robot body 1 is a trajectory that collides with the obstacle S. 17 is stored.
[0035]
FIG. 5 shows conditions that become singular points when the robot body 1 controls the position and orientation of the arm. For example, when shifting from the state shown in FIG. 5A to a condition where the upper arm portion 10 and the wrist portion 11 are in a straight line as shown in FIG. The rotation coordinate data will exist infinitely (the axis provided between the arms is the one in which the rotation axis is formed in the rotation direction of the cylindrical rotating body as shown in FIG. The solution is an indefinite singularity. In this case, the control by the visual servo is interrupted, and the subsequent control becomes impossible.
[0036]
Corresponding to such a case, as the data of the movement prohibition area, the condition in this state is stored in the restriction data storage unit 17 as the trajectory restriction data. Thereby, at the time of actual control, when the singular point is approached, the gain as control data for setting the arm movement amount is changed by the control device 2 so as to avoid the singular point, for example, the gain is temporarily made zero. Set.
[0037]
In the case shown in FIG. 6, when the workpiece W as shown in FIG. 6A rotates, the camera 5 is controlled to rotate following this. However, the fact that the camera 5 takes the position and orientation as shown in FIG. 4B means that the upper arm portion 10 of the robot body 1 rotates accordingly. Actually, however, the camera 5 is provided with the cable L as shown in FIG. 5C, and therefore, it is necessary to limit the rotation amount so that the cable L is not disconnected.
[0038]
Corresponding to such a case, the amount of rotation of the upper arm unit 10 is limited, and this is stored and set in the limit data storage unit 17 as limit data. For example, when the rotation amount reaches a predetermined value or more, it is possible to stop further rotation by setting the gain of the rotation angle to zero.
[0039]
The restriction data described above can be dealt with by storing in advance in the storage unit 17 prior to the execution of visual servoing. In addition, for example, an obstacle S such as a distance sensor is detected. It is also possible to provide a sensor capable of generating limit data by inputting the detection signal and using the control device 2.
[0040]
Further, the control device 2 may be provided with a communication device, and such movement restriction data may be fetched from outside through communication and set in the storage unit 17. Thereby, it becomes possible to register the restriction data easily and quickly by receiving the restriction data created in advance according to the work environment of the robot body 1.
[0041]
As described above, when moving to the area where the restriction data is registered, the position and orientation are controlled so as to avoid this, but at this time, the control by the visual servo is surely performed in the avoided place. Therefore, a reference image used when avoiding the restricted data area can be separately registered as a switching model. As a result, even when the avoidance track is moved, the workpiece W can be reliably recognized and the control by the visual servo can be continued.
[0042]
Next, the case of controlling the position and orientation by visual servo based on the registered data list will be described with reference to the flowchart of FIG. The control device 2 moves the robot body 1 to a position where the detection operation is started (step S1), and captures and acquires an image of the workpiece W with the camera 5 (step S2). It is determined whether or not the workpiece W exists in the photographed image (step S3). If “NO”, steps S2 and S3 are repeatedly executed.
[0043]
If “YES” is determined in the step S3, the control device 2 starts control of the position and orientation by the visual servo, and drives and controls the robot body 1 so as to move the inspection position to the target position. (Step S4). Thereafter, the control device 2 determines whether or not the robot body 1 has reached the target position at an appropriate timing based on the image taken by the camera 5 (step S5). The position and distance are calculated, and the detection signal of the sensor is input (step S6).
[0044]
Next, when the trajectory from the current position of the robot body 1 to the target position is generated based on the visual servo control method, the control device 2 determines whether the generated trajectory corresponds to the restriction data. (Step S7). Here, in the case of “YES”, that is, when there is a restriction, it is determined whether or not the model that is the reference image of the work W needs to be switched (step S8), and in the case of “YES”, the model is switched. (Step S9), and thereafter, the limiting function is turned on (step S10). Accordingly, the control device 2 performs movement control by correcting the trajectory based on the position and orientation control of the visual servo based on the restriction function, that is, the restriction data stored in the restriction data storage unit 17.
[0045]
In the above case, for example, when the generated trajectory corresponds to the restriction data, the model needs to be switched corresponds to the case where there is a possibility of colliding with the obstacle S shown in FIG. Thus, in order to control the trajectory to bypass the obstacle S and follow the movement of the work W, the reference image on the bypassed trajectory is switched and used as a model.
[0046]
In addition, when the restriction function is turned on corresponding to the restriction on the operation of the robot body 1 as shown in FIGS. 5 and 6, the robot body 1 is controlled so as to avoid the state. Since the relative positional relationship between the camera 5 and the workpiece W does not change, this usually corresponds to a case where switching between models is not required.
[0047]
In the control of the position and orientation of the robot main body 1, when the avoidance operation is performed with the restriction function turned on, information such as the avoidance direction and the avoidance position is set and stored in advance so that a reliable avoidance measure can be taken. Alternatively, a determination processing program in that case can be set in advance, and a direction not to lose sight of the workpiece W can be selected and moved.
[0048]
In that case, it is an effective means to capture the detection output of the sensor as an element in determining the direction. In order not to lose sight, such a direction may be stored as a table and determined by referring to the table at the time of control.
[0049]
Furthermore, as one of the criteria for setting the direction when such avoidance is performed, if there is a danger in the movement of the robot body 1 or if there is a worker in the work environment, the worker It is also a useful method to provide conditions such as turning away from the camera.
[0050]
In this way, the position / orientation control by the visual servo in a state in which the restriction function by the control device 2 is turned on is advanced, and the process returns to step S4 to repeat the above-described processing. If it is determined in step S7 that there is no restriction, the restriction function is turned off (step S11), the process returns to the normal visual servo position and orientation control, and the processes in and after step S4 are repeatedly executed. .
[0051]
When the control device 2 receives the restriction data from the outside via the communication device while executing the position / orientation control as described above, this is taken into the restriction data storage unit 17 by a control program (not shown) and prohibited from moving. Registered as area data. Therefore, when a trajectory corresponding to the newly registered restriction data is generated during the control operation, the control device 2 performs position and orientation control so as to correct the trajectory and avoid the movement prohibition area. As a result, the position and orientation can be controlled while flexibly responding to changes in the working environment of the robot body 1 and avoiding accidents such as collisions.
[0052]
Thereafter, when the workpiece W of the image captured by the camera 5 coincides with the workpiece W of the reference image, the control device 2 determines that the position and orientation of the robot body 1 has reached the target position. “YES” is determined, and the position / orientation control by the visual servo is ended.
[0053]
According to this embodiment, the following effects can be obtained. That is, when controlling the position and orientation by visual servo, the restriction on the work environment and the control of the robot body 1 is stored as the restriction data in the restriction data storage unit 17, and the controller 2 generates the trajectory of the robot body 1. When the data falls under the restriction data, the position and orientation is controlled by avoiding this, so even if there is a risk of a malfunction such as a collision or inability to control, the work W can be reliably avoided by avoiding this. On the other hand, the position and orientation can be controlled up to the target position indicated by the reference image.
[0054]
In the above-described case, by setting an obstacle or the like of the work environment in which the robot body 1 moves as the movement prohibited area, the position and orientation can be reliably controlled following the workpiece W while avoiding these. In addition, as the uncontrollable condition, the limit value of the rotation amount of the arm of the robot body 1 can be set, or singular points can be set to avoid these. Thus, it is possible to control the position and orientation while avoiding the problems that occur.
[0055]
And according to this embodiment, the method of setting the movement prohibition area in advance, the method of detecting by a sensor and automatically setting it, the communication device etc. is provided and received as restriction data from the outside This method can be applied, and the position and orientation can be controlled quickly and reliably by visual servoing in response to various factors such as work environment and internal structure and their changes flexibly. Will be able to.
[0056]
The present invention is not limited to the above embodiment, and can be modified or expanded as follows.
That is, in the above-described embodiment, the trajectory to be prohibited as the movement prohibition area is stored as the restriction data. However, the present invention is not limited to this, and it can be set as the prohibition area itself and can also be set as the prohibition direction. Furthermore, the constraint conditions for controlling the position and orientation of the robot body 1 can be set in various forms.
[0057]
Also, in setting the movement prohibition area, the judgment is made before it actually becomes uncontrollable, so that it can be controlled so as to be surely avoided, so that it enters the danger area within the controllable range. When this is expected, this may be estimated and the model switched or an approach to avoid the prohibited area may be indicated.
[Brief description of the drawings]
FIG. 1 is an overall schematic configuration diagram showing an embodiment of the present invention.
FIG. 2 is a functional block configuration diagram of a control device.
FIG. 3 is a flowchart of a position / orientation control program;
FIG. 4 is an explanatory diagram for avoiding a collision with an obstacle.
FIG. 5 is an explanatory diagram of singular points of robot arm drive.
FIG. 6 is an explanatory diagram for limiting the amount of rotation of a robot arm.
FIG. 7 is a diagram for explaining a defect in the prior art
[Explanation of symbols]
1 is a robot body, 2 is a control device, 3 is a robot controller, 4 is a visual recognition device, 5 is a camera, 14 is a visual recognition unit, 15 is a model data storage unit, 16 is a trajectory generation unit, and 17 is a restriction data storage unit , 18 is a trajectory control unit.

Claims (10)

The target object is photographed as a reference image in advance from a reference position by a camera attached to the control target device and registered, and the control is performed so that the image of the target object photographed by the camera is in the same state as the reference image. In a visual recognition device comprising a control means for controlling the position and orientation of the target device,
In the position / orientation control by the control unit, a storage unit is provided for setting and storing the position / orientation region of the control target device that hinders control of the position and orientation of the control target device as a movement prohibition region;
The control means compares the image of the object photographed by the camera with the reference image.
The trajectory to the movement destination in the position and orientation control of the control target apparatus is calculated, and when the calculated trajectory passes through the movement prohibition area set in the storage unit, the movement prohibition area is bypassed. A visual recognition device configured to move along a trajectory . The visual recognition device according to claim 1,
Wherein the storage means, a visual recognition device, characterized in that a region including the position of the obstacle present within a movement range of the control target device is stored as the moving prohibited area. The visual recognition device according to claim 1 or 2,
An area including a position and orientation corresponding to an uncontrollable condition in which movement is difficult during position and orientation control is stored as the movement prohibition region in the storage unit from the structural characteristics of the control target device. Visual recognition device. The visual recognition apparatus according to claim 3.
The control target device is a multi-joint robot having a singular point of the arm,
The visual recognition apparatus according to claim 1, wherein a posture of a singular point of the arm of the articulated robot is set as the movement prohibited region in the storage unit as the uncontrollable condition. The visual recognition device according to claim 3 or 4,
The control target device is a robot with a rotatable arm,
The visual recognition apparatus according to claim 1, wherein a range of limitation on the amount of rotation of the robot arm is set as the movement prohibition area in the storage unit as the uncontrollable condition. The visual recognition device according to any one of claims 1 to 5,
The visual recognition apparatus according to claim 1, wherein the movement prohibition area is registered in the storage means as information stored and set in the storage means in advance. The visual recognition device according to any one of claims 1 to 6,
The control target device is provided with a sensor that detects a distance for recognizing an area where an obstacle that collides due to movement for position and orientation control exists ,
The control means recognizes the detected area as the movement prohibition area and stores the detected area when the sensor detects the obstacle existence area when controlling the position and orientation of the device to be controlled. A visual recognition device characterized by having a memory setting. The visual recognition device according to any one of claims 1 to 7,
Providing a communication means capable of receiving restriction data transmitted from the outside as information indicating a position and orientation that interferes with controlling the position and orientation of the control target device;
The visual recognition apparatus characterized in that the control means recognizes the information on the movement prohibited area based on the information on the restriction data received by the communication means, and stores the information in the storage means. The visual recognition device according to any one of claims 1 to 8,
Wherein the storage means, a visual recognition device, characterized in that the path-constrained data providing restrictions on orbits generated corresponding to the movement of the position and orientation of the control target device is stored as the moving prohibited area. The visual recognition device according to claim 9, wherein
The control means, when restricting the trajectory of the control target device based on the trajectory restriction data stored as the movement prohibition area in the storage means, when approaching a singular point where trajectory control becomes impossible, visual recognition apparatus characterized by controlling the trajectory by change sets the gain for controlling the respective sections so as to avoid points.

Images