JPH0425583B2 - - Google Patents

Description

[Detailed Description of the Invention] [Field of Application of the Invention] The present invention relates to the presence or absence of a target object necessary for performing various operations such as assembly, inspection, and adjustment by a robot.
The robot recognizes the position, posture, and spatial situation within the work area based on the base that fixes the robot, and further recognizes the position and posture of the target object based on the movable parts of the robot, making it possible to control the robot using complete visual feedback. The present invention relates to a robot control device that realizes control so that the robot can perform various tasks such as highly accurate assembly, inspection, and adjustment on work objects.

[Background of the invention]

To recognize the position and orientation of an object using robot vision, an image detector 3 is placed on a work table 1 as shown in FIG. To detect. Then, the position and orientation of the object are recognized, and the robot arm 2 operates according to that information. In this method, the plane position and orientation of an object on the table plane can be recognized, but the three-dimensional shape cannot be recognized. For example, it is not possible to approach objects piled up on a table by tilting the robot arm so as to match the grip surface of the object according to the inclination information of each object. Furthermore, unless there is a clear contrast (difference in shading) between the object and the work table, the position and orientation on the table plane cannot be determined.

On the other hand, there are robots with visual devices attached to their hands that provide feedback. FIG. 2 shows an example of this, in which a welding line 5 to be welded by a welding head 7 is tracked. This involves attaching a slit light projector 6 to the tip of the robot arm 2 and projecting a slit light 8 onto the welding line. The image is detected diagonally by an image detector 3 attached to the tip of the robot arm, the position of the weld line is determined, and the robot arm is controlled so that the deviation from the target value is zero. However, this method can only detect the position of the weld line, that is, the position of the groove-shaped object, and cannot recognize the three-dimensional position and posture of the object and perform operations such as assembly, inspection, and adjustment. .

[Purpose of the invention]

An object of the present invention is to correct the position and orientation of the robot by recognizing the position and orientation of the target object in the coordinate system of the robot, regardless of the orientation and shape of the target object, in order to solve the problems of the prior art described above. A robot that realizes robot control using complete visual feedback so that the robot can perform various tasks such as high-precision assembly, inspection, and adjustment of the target object without having the robot perform extremely complicated search operations. The purpose is to provide a control device.

[Summary of the invention]

In order to achieve the above object, the present invention provides a robot having a movable member, and information on the distance from a point determined with respect to a base portion to which the robot is fixed to each point in the entire working area of the robot. A distance image detection means for detecting the distance image signal as a two-dimensional distance image signal at once, and a distance image detection means for two-dimensionally differentiating the distance image signal detected by the distance image detection means, Based on the signal, a two-dimensional surrounding area is extracted to find the workpiece, and the surface of the workpiece is detected based on the distance image signal detected by the distance image detection means in the extracted area. a first image processing means for determining a line direction and position; a robot image detector fixed to a movable member of the robot; and a first image processing means for determining a workpiece determined by the image processing means. a robot control means for controlling the position and attitude of the robot by driving and controlling a movable part of the robot so as to have a specific angle and distance relationship with respect to the normal direction and position of the surface of the robot; a second image processing means for calculating the position and orientation of the workpiece based on the robot image detector from the image signal thus detected; and a robot image detector calculated from the second image processing means. This robot control device is characterized by comprising a correction control means for correcting the position and orientation of the robot by driving and controlling the movable parts of the robot based on the position and orientation of the work object as a reference.

[Embodiments of the invention]

Hereinafter, the present invention will be explained in detail with reference to the drawings.

First, the processing contents in the present invention will be explained. As mentioned above, the processing consists of the first step of detecting a distance image of the entire work object from a fixed point in the robot's coordinate system, discovering the target object, and recognizing its position and orientation; After the robot arm approaches the target object using the results of
It consists of the following stages (Figure 13).

In the distance image 9 detected in the first step, as shown in FIG. Vij=Klij+C...(1) It is an image with a linear relationship where K: non-zero constant and C: constant. Therefore, especially when K is a negative number, Vij can be regarded as representing the height of each point on the object surface.
Now, as shown in Fig. 4, if we define a situation in which several objects are folded on top of each other and are captured by a distance image detector from directly above, then the fifth
A distance image 9 as shown in the figure is detected. As is clear from FIG. 5, since the distance to the detector generally changes rapidly at the boundary line between objects or between objects and the background, the value of the distance image also changes rapidly. In distance images, such boundaries are called jump edges. If this change is detected using two-dimensional differentiation of the image, the result will be as shown in FIG. 6. Several methods are known for two-dimensional differentiation of images, but here we will use the point (i,
The differential value dij of j) is dij=max(|Vij−Vi−1, j−1|, |Vi−1, j−Vi, j−1|) i, j=1, 2, 3… ……( 2) The calculation is as follows. However, (i, j) are orthogonal coordinates of a plane perpendicular to the distance direction between the detector and the object. When the obtained differential image is binarized using a certain fixed threshold value, a jump edge image as shown in FIG. 8 is obtained. In the figure, the black line represents the jump edge 12, which corresponds to the outline of the object as described above. Therefore, the area surrounded by the jump edge 12 corresponds to each object, and by separating and extracting each area on the image, the objects can be separated on the image.

From the region R corresponding to each object separated and extracted as described above, its area S and the average height H of the region, i.e. For S ₁ ≦S≦S ₂ H ₁ ≦H≦H ₂ . . . (4) A region that meets the criterion is extracted as a region of the target object. S ₁ and S ₂ are the upper and lower limits of the area variation range when the target object is projected onto a certain plane, and H ₁ and H ₂ are the upper limits of the height at which the target object is expected to exist. Each of these is a lower limit value that is set in advance. The shape of the area on the image of the target object separated in this way generally does not have a constant shape. That is, this shape is the two-dimensional shape of the target object projected onto a certain plane. For example, if the three-dimensional shape of the target object is a rectangular parallelepiped, it may be rectangular or rectangular depending on its orientation as shown in Figure 9. It becomes a hexagon. In the present invention, the three-dimensional shape and position of the target object,
In order to recognize the pose, the target object is assumed to be composed of several planes, and the intersection lines between the planes, that is, the ridge lines, are detected. The crest line is a point on the distance image where the rate of change in value changes, as shown in Figure 10.
That is, it corresponds to a point where the second-order differential value of the image has an extreme value. Such a group of lines is called a half edge 13 in a distance image. There are various methods for second-order differentiation of images, but here we will use The second-order differential value rij is used. Here, n is an integer of 2 or more, and the value of rij corresponds to a change in the inclination angle of the distance image value Vij of the point of interest (i, j). Next, the roof edge 13 is detected by binarizing the second-order differential image calculated using equation (5) according to a certain threshold value. The calculation shown in equation (5) may be applied to the entire detected range image, but
Performing calculations only on the region of the target object that was previously separated and extracted leads to a reduction in processing time.

An image composed of jump edges 12 and half edges 13 of the target object detected through the above-described processing is as shown in FIG. 11, for example, in the case of the distance image of the tilted rectangular parallelepiped shown in FIG. 10. Since the regions surrounded by the jump edges 12 and half edges 13 correspond to the individual faces that make up the object, by separating the regions on the image and extracting the region 14 with the largest area, the main parts that make up the object can be extracted. One of the faces 15 can be taken out. Next, plane approximation is performed for this region. In other words, the plane equation to be found is t ₁ x + t ₂ y + t ₃ = z ...(6), and the distance image of m points in the area is
If the xyz coordinates are (ak ₁ , ak ₂ , bk) k = 1, 2..., m, then t ₁ , t ₂ , t ₃ in equation (6) are t as a matrix t whose row is (tk). It is calculated as = (A ^T A) ^-1 A ^T b ...(7). Here, A is a matrix whose rows are (ak ₁ , ak ₂ , 1.0), b is a matrix whose rows are (bk), and T
represents the transposed matrix, ^- ₁ represents the inverse matrix. Also, the xyz coordinates (ak ₁ , ak ₂ , bk) are the coordinates (i,
j) and the distance image value Vij are subjected to linear transformation and obtained as the following equation (8).

ak ₁ = P ₁ .i _{+ q 1} ak ₂ = _{P 2 .} , p ₂ , p ₃ , q ₁ , q ₂ , and q ₃ are constants determined based on the detection resolution of the detector, etc.

From the approximation formula (6) for the main plane 15 of the target object obtained in this way, the three-dimensional posture of this object is determined by the angle between the normal direction 16 of the main plane and the z-axis and the normal direction. Using the angle θ between the projection of the line onto the xy plane and the x-axis, It is expressed as θ=tan ^-1 t ₂ /t ₁ (10) (Figure 12). Once the direction of the normal line is determined, the shape of the main plane seen from this direction can be determined by simple coordinate transformation, and from the transformed shape, the shape of the target object can be determined using conventional planar image processing technology. , posture recognition is also possible.

Further, the position of the target object is determined by the following equation (11) as the center of gravity position (x ₀ , y ₀ , z ₀ ) of the main plane.

x ₀ =k ₁ y ₀ =k ₂ z ₀ =k (11) where - represents the average within the area. Further, the average in this case is not limited to m points within the region, but can be calculated over the entire region to obtain a more accurate one.

Now, the processing results from the first stage explained above, that is, the position of the center of gravity of the main planes of the target object (x ₀ , y ₀ , z ₀ ) and the direction of the normal, θ, are determined by the coordinate system of the detection system. Since this coordinate system is generally different from the coordinate system of the robot arm, it is not possible to directly control the robot arm using these values. For this purpose, the coordinates are transformed into the coordinate system of the robot arm. Depending on the control system of the robot arm, various coordinate systems such as a rectangular coordinate system, a cylindrical coordinate system, and a polar coordinate system can be considered, and conversion to these coordinate systems is well known from the formulas of linear algebra.

Although the calculation methods such as equations (2), (5), and (7) are limited in each process of the first stage explained above, other calculation methods having similar functions may be used. Of course.

Next, the robot arm is controlled so that the image detector installed on the robot arm can detect the object from the direction normal to the center of gravity of the main plane of the object.

Thereafter, an image is detected from above the robot arm, the positional relationship between the robot arm and the target object is detected more accurately, and the process moves to the second stage, which leads to accurate work of the robot arm. The image detector 20 on the robot arm uses one with higher resolution than the range image detector used in the first stage, but since the target object has been discovered in the first stage, its detection field of view can be small. . As in the first stage, the detection images include those that detect distance images and perform similar processing, those that detect and process two-dimensional grayscale images, those that process binary images, and the optical cutting method using slit light. There are various methods that can be considered, such as a method that applies the above, but any of them may be used.

Next, a specific example of an apparatus that implements the processing contents described above will be described. FIG. 13 shows an outline of the overall configuration. As shown in the figure, the apparatus includes a robot system consisting of a robot arm 17 and a robot control device 18, a visual system consisting of a distance image detector 19, an image detector 20 on the robot arm, and an image processor 21.

First, the robot system will be explained using FIG. 13. 17 is an industrial robot, for example, an articulated robot having five degrees of freedom. The industrial robot 17 has a rotating base 17a that rotates around a vertical axis with respect to the base 22, an upper arm 17b that rotates around a horizontal axis 23a, a forearm 17c that rotates around a horizontal axis 23b, and a horizontal arm 17c that rotates around a horizontal axis 23b. The wrist 17d rotates around an axis 23c, and further rotates around an axis perpendicular to the horizontal axis 23c.
A hand mechanism 24 with fingers (chucks) 25 is attached to the wrist 17d. The image detector 20 attached to the robot arm may be attached to any of these locations, but it is preferably attached to the hand mechanism 24 in order to operate the robot and detect images from the most appropriate direction. . 18 is a robot control device that controls the robot.

Next, the robot control device 18 will be explained with reference to FIG. The robot control device 18 has a control unit 26 for controlling the articulated robot 17 having five degrees of freedom and a robot. Two.
In order to operate or move along a predetermined trajectory based on the speed programmed at the point, the teaching unit 27 is configured to teach information on a pre-programmed trajectory and speed. The control unit 26 and the robot mechanism 17 constitute a position control system. This position control system counts the output pulses generated by the pulse encoder PE connected to the actuator M with a counter 28 and feeds them back to the control unit 26 to obtain a target predetermined by a microprocessor 29. A digital signal that is different from the value or a desired coordinate value given from the outside is detected, and this digital signal is sent to the D/A converter 30.
The configuration is such that the signal is converted into an analog signal and the actuator M is driven.

The drive circuit 31 is a circuit that drives the actuator M based on a speed signal from a tachogenerator TG connected to the actuator M and an analog signal from the D/A converter 30. Serial interface 32 is for connecting to teaching unit 27. The ROM 33 is a memory that stores programs for operating the robot. The RAM 34 stores information from the teaching operation performed using the teaching unit 27 or the information from the interface 3.
The operation trajectory of the robot's hand machine 24, which is the result of interpolation calculation performed by the calculation section 36 on the operation information input from the image processor 29 via the image processor 5, is stored. 37 is a bus line.

Robot hand mechanism 24 stored in RAM 34
The position data is read out by the microprocessor 29, coordinates are transformed into rotational displacements θ ₁ , θ ₂ , ... θ 5 detected by the counter 28, and the coordinates are converted into rotational displacements θ 1 , θ 2 , ... θ ₅ detected by the counter 28 . The hand mechanism 24 of the robot is driven to the object position).

Although the above description of the robot system is limited to a robot having five degrees of freedom, a robot having other degrees of freedom, for example, six degrees of freedom, can be constructed in the same way. In the present invention, there are no particular limitations on the degree of freedom.

Next, the visual system, which is another component of this embodiment, will be explained.

An embodiment of the distance image detector 19 is shown in FIG. As shown in the figure, the detector includes a slit light source 3 that emits slit light 38 from above the object 11.
9, an image pickup device 40 that detects an optical image of the intersection line of the slit light 38 and the object 11, that is, the slit bright line from a diagonal direction, and a feed device 41 that drives at a constant speed in the horizontal direction while maintaining their positional relationship. , and an intersection line extraction device 42 that separates and extracts the slit bright line detected by the imager 40 from the image and outputs its shape as a waveform signal. This operation will be explained using FIGS. 16 to 20. It is assumed that the slit light 38 hits the object 11 at the position shown in FIG. 16. Then the imager 40
The image detected in this step is, for example, as shown in FIG. Vertical lines on this image, e.g.
The change in brightness along AB is shown in Figure 18. By sequentially moving line AB in the i direction and extracting the position of the brightest point (c in Figure 18) along this line AB, the shape of the slit bright line can be changed into a waveform as shown in Figure 19. It can be taken out as a signal. This shape indicates the cross-sectional shape of the object 11. The separation and extraction of the above waveform signals is as follows:
This is performed by the optical cutting line extraction device 42 shown in FIG. A specific example of this optical cutting line extraction device 42 is disclosed in, for example, Japanese Patent Laid-Open No. 70407/1983.

That is, as shown in FIG. 26, the optical cutting line extraction device 42
are the video signal V from the image detector, the set values V ₁ , V ₂
(V ₁ > V ₂ ), and the trigger signal Hd for each scan of the image detector (horizontal synchronization signal in the case of a TV camera) are input, and the maximum value position detection circuit 57, center position detection circuit 58, and selection circuit 59 configured.

As illustrated in FIG. 27a, the function first detects the maximum value position when the maximum value Vmax of the video signal is within the range of the first and second set values _V1 and _V2 . The y position Y can be determined by the circuit 7.
Further, when Vmax is larger than _V1 (in the case of FIG. 27b), the center position detection circuit 58 first detects that Vmax is
The y position Y ₁ where V = V ₁ is obtained, and finally the y position Y ₂ where V = V ₁ is determined, and further, the center position thereof is determined by setting Y' = (Y ₁ + Y ₂ )/2. During one scan, if V becomes equal to or greater than V ₁ in the circuit 58, the selection circuit 59 selects the output of the circuit 58, while V
If the value does not exceed V ₁ , the selection circuit 59 outputs the output of the circuit 57 as the position of each optical cutting line. Further, if V does not exceed V ₂ during one scan, the selection circuit 59 outputs zero or an undefined output.

FIG. 28 shows the embodiment of FIG. 26 in more detail. The contents of the operation will be explained in more detail using FIG. 28.

The maximum position detection circuit 57 includes a peak holder 60,
It is composed of a comparator 61, a one-shot circuit 62, an AND circuit 63, a register 64, a comparator 66, and a flip-flop 67. on the other hand,
The center position detection circuit 58 includes a comparator 68, a flip-flop 69, and one-shot circuits 70, 7.
1, _Y1 register 72 _Y2 register 73 and AND circuit 74. It also has a counter 65 to know the y position at each moment. In the maximum position detection circuit 57, the video signal V enters a peak holder 60, which holds the peak value of V up to a certain moment during one scan. Comparator 61 compares this held peak value with V at that moment, and outputs a signal to one shot circuit 62 when V becomes larger than the held peak value by a predetermined value or more. On the other hand, V is compared with V ₂ in a comparator 66, and outputs when V≧V ₂ . By ANDing this with the output of the one-shot circuit 62 in an AND circuit 63, it is possible to detect the moment when V≧V ₂ and V reaches its peak. This is used as a load signal for the register 64, and the instantaneous value of the counter 65, which is reset by the trigger signal Hd, is stored in the register 64.
Therefore, at the end of one scan, the video signal V holds the y-coordinate position Y where the maximum value Vmax is shown.

In the center position detection circuit 58, the video signal V is compared with a set value _V1 by a comparator 68, and its output is input to the set side S of a flip-flop 69. Note that the flip-flop 69 is reset by the trigger signal Hd. Therefore, the one-shot circuit 70 initially receives V in one scan.
Detect the moment when ≧V ₁ . Add this to register 7
By using it as a load signal of 2, the register 72 holds the position Y ₁ on the y coordinate where V≧V ₁ first. On the other hand, the one-shot circuit 71 detects the instant when V≧V ₁ no longer holds, and if this is used as a load signal for the register 73, the register 73 will contain y when V≧V ₁ no longer holds at the end of one scan. The coordinate position Y ₂ is held. 74 is an average value circuit that calculates Y'=(Y ₁ +Y ₂ )/2.

The output of flip-flop 69 can be used to detect whether the condition V≧V ₁ existed during one scan. Furthermore, since the flip-flop 67 uses the trigger signal Hd as a reset signal and the output of the comparator 66 as a set signal, it can be used to detect whether the case of V≧V ₂ exists during one scan. Therefore, the selection circuit 59 uses these as condition signals,
If V≧V ₁ exists, the hold value Y′ of 74 is
When V ₂ <V≦V ₁ , if there is no case in which the hold value Y of 64 is V≧V ₂ , a signal representing zero or indeterminate is output.

In this embodiment, the details of the image detectors 40 and 42 are not mentioned, but they may include a TV camera secondary solid-state image sensor, a combination of a one-dimensional solid-state image sensor and an optical scanning method, a photomultiplier tube, etc. It may be any combination of a light receiver and an optical scanning method.

Furthermore, in the embodiments of FIGS. 26 and 28,
Although illustrated as having dedicated hardware for achieving each means, all of these means are micro. It can be processed using a program using a computer.

Furthermore, by gradually moving the position of the slit light 38 and the imaging position using the feeding device 41 at a constant speed and successively extracting waveform signals in the shape of the slit bright line, a distance image is obtained as a whole as shown in FIG. 9 is obtained.

The imager 40 in this embodiment may be any two-dimensional image detector such as a TV camera or a combination of a linear sensor and a galvano mirror.According to this embodiment, distance images can be captured with high precision with a relatively simple configuration. Can be detected.

FIG. 21 shows another embodiment of the distance image detector 19. As shown in the figure, this embodiment has a structure in which two sets of the detector imager and the optical cutting line extraction device shown in FIG. 15 are combined. That is, two imagers 40a and 40b are provided at symmetrical positions with respect to the plane formed by the slit light 38, and the feeding device 4
1 to move the slit light source 39. The method for extracting the optically cut waveform is exactly the same as in the previous example, but in this embodiment, the obtained waveform signals are combined 43 so as to compensate for the cut portions of the waveforms to generate a distance image. Here, the broken part of the waveform is the 22nd
As shown in the figure, the slit bright line corresponds to a portion 44 that cannot be detected by one of the imagers, and becomes a kind of shadow in the distance image. In this embodiment, as described above, since waveform signals detected from two directions are combined, there is an effect that a distance image with fewer shadows can be obtained.

In addition, as a distance image detector, there is a method that generates a distance image by irradiating a pulsed laser beam onto corresponding points and measuring the flight time of the reflected light, and a method that generates a distance image by applying high frequency amplitude modulation to the laser beam. There is a known method of generating a distance image by emitting light and measuring the phase delay of the high frequency amplitude of the reflected light, but these may also be used. In these methods,
Since the distance between each point on the object is measured using a laser spot, a mechanism for scanning the spot two-dimensionally, such as two sets of galvano mirrors, is required. Since the reflected light can be detected from directly above, it has the advantage of generating a distance image with no invisible parts, that is, no blind spots or shadows.

Next, an embodiment of the image detector 20 attached to the robot arm will be described.

FIG. 23 shows an embodiment of a distance image detector as an image detector 20 attached to a robot arm. The TV camera 45 detects a light cut waveform caused by the slit light 38 projected from the slit light source 39. This detector is attached to the robot's hand mechanism 24, and a distance image can be generated by extracting a light section line and capturing a waveform signal while moving the hand. The distance image can be processed in exactly the same way as the distance image attached to a fixed position described above. Note that details of this embodiment and other embodiments are as follows. That is, 77 is an industrial robot, for example, an articulated robot having five degrees of freedom. The industrial robot 17 has a rotating base 17a that rotates around a vertical axis with respect to a base 22, and a horizontal axis 23.
The upper arm 17b rotates around a, the forearm 17c rotates around a horizontal axis 23b, the forearm 17c rotates around a horizontal axis 23c, and the upper arm 17c rotates around a horizontal axis 23c.
and a wrist 17d that rotates around an axis perpendicular to c. A hand mechanism 24 with fingers (chucks) 25 is attached to the wrist 17d. This hand mechanism 24 is equipped with a main body shape detector 76. The main body shape detector 76 includes a light cutting head 77, a linear movement mechanism 79 that scans the light cutting head 77 in the y-axis direction, a motor 80 that drives the linear movement mechanism 79, and a linear movement mechanism 79.
9, and a displacement detector 82 such as a rotary encoder for detecting the amount of scanning of the optical cutting head 77 from the reference position.

Although FIG. 29 only depicts a feed screw and a nut as the linear movement mechanism 79, there is actually a sliding mechanism.

83 is a motor control circuit that drives the motor 80 at a constant speed. 42 inputs the two-dimensional video signal obtained from the image detector 78 and
This is an optical cutting line extraction circuit that extracts optical cutting lines as described in No. 70407. Reference numeral 21 denotes an image processor which detects the position and inclination of the object closest to the robot and which the robot wishes to work on. A robot control device 18 controls the robot 17.

FIG. 23 shows the optical cutting head 77 in the embodiment of FIG. 29 in more detail. The slit floodlight 39 is a linear filament lamp 8.
4, a slit 85, and a cylindrical lens 86. The image detector 78 includes an imaging lens 87 and a TV camera 45. 88 in the figure is a solid-state TV
This figure shows the chip of the two-dimensional array sensor inside the camera.

The slit projector 39 uses a cylindrical lens 86 to turn linear light emitted from the slit 8s out of the light generated by the lamp 84 into parallel light rays and irradiates the front thereof. 38 shown in FIG. 23 indicates the surface of the emitted slit light. The image detector 78 is connected to the holding section 20
The optical axis is fixed at an angle so that it intersects the slit optical surface 38a obliquely (at an angle α).
The trapezoidal surface in FIG. 23 38b is the slit light surface 38a.
In the middle, the field of view detected by the image detector 78 is shown.
When there is an object within this field of view, a bright line 38b is generated on the object surface as illustrated in FIG. 30, and an image of this bright line is detected by an image detector. Figure 30 shows a connector part 90 with wires as an example of robot work.
The figure shows the work of detecting the position and posture of the robot, grasping it, and inserting it into the pin 92 on the board 91. 31st
The figure shows an image detected by the image detector 78 shown in FIG. 23, which is brightly detected as a slit bright line image. As is clear from the mechanical relationship between the slit projector 39 and the image detector 78 in FIG. 23, this screen has a perspective relationship in which the top of the screen is far away and the bottom is close. The light cutting line extraction circuit 42 extracts light cutting lines using the slit bright line image shown in FIG. 31 as the distance from the screen to the slit bright line. FIG. 32 shows an example of the processing. (Details are disclosed in JP-A No. 56-70407.) No. 32
Figure a shows an input image input from the image detector 78 to the optical cutting line extraction circuit 42. For example, the video signal of one scanning line xi in the vertical direction obtained from the image detector 78 is as shown in FIG. 32b. Threshold from this
Compare V ₁ and the video signal, and find the center position Zi of the intersection of V ₁
seek. By determining and outputting Zi for each Xi, an optical section line (waveform data) can be obtained as shown in FIG. 32c. Note that, in addition to finding the center using the threshold value processing shown here, the optical cutting extraction may also be performed by detecting the peak position. Further, in this example, the far side of the Z-axis coordinate is set as the origin, but the near side may be set as the origin.
It is also easily possible to perform coordinate transformation in the Z-axis direction according to the perspective law. Further, although this processing can be performed at high speed using an electric circuit, it may also be performed entirely by software processing.

By performing the above processing while moving the light cutting head using the motor 80, a distance image can be obtained. An example is shown in FIG. FIG. 33 is an example in which the optical cutting lines are thinned out and displayed. i.e. xy
When viewed as a planar image, the brightness Z corresponds to the distance S from the light cutting head 3. The brighter the image, the closer the image. (In Figure 34, Z ₂ = S ₂ sin α,
It is expressed as Z ₁ =S ₁ sinα. ) The movement of the head, that is, the feeding in the y-axis direction, is performed by a constant speed motor, and the optical cutting line extraction circuit 42 samples the optical cutting line at regular time intervals using a sampling signal 92 obtained from the image processor 21. While being fed by the pulse motor 80, the optical cutting line is sampled at constant pulse intervals (at every interval of movement in the Z direction by a constant distance P). DC motor 8
0 and the encoder 82 and sampled every time a certain amount of movement is performed. The number of optical cutting lines to be detected may be selected from two or more depending on the object of robot vision work. The same goes for Pituchi. These are controlled by the image processor 21. The processing content of the distance image input to the image processor 21 may also be selected depending on the object of robot vision work. As an example, regarding the work target shown in FIG. 30, the obtained distance image is that shown in FIG. ) is obtained from the optical cutting line extraction circuit 42 and stored in the image memory RAM 51a via the parallel interface 50b. For example, the microprocessor 48 of the image processor 21 processes adjacent light cutting lines as shown in FIG.
Distance Zi−1j−1, Zi for xi−1 and xi+1 (for 3×3 picture elements) for yi−1 and yi+1
+1, j-1, Zi-1, j+1, Zi+1, j+1
Read out, Zi, j=|Zi−1, j−1−Zi+1, j+1|+
|Zi+1, j-1-Zi-1, j+1|/2 If this value is larger than a certain large reference value, there is a jump edge (boundary line between the object and the background), so the above values of Zi, j RAM
62 in a separate area. Furthermore, the microprocessor 48 performs segmentation on a closed area surrounded by a solid line (jump edge) Ej as shown in FIG. Then, the area of A ₁ and A ₂ and the height of the center of gravity are determined for each closed region, and the target object whose center of gravity is within the specified range and has the highest height is extracted.

Furthermore, for example, a plane area surrounded by the roof edge Er and jump edge Ej shown by dotted lines in FIG. 35 is extracted, the separated and extracted plane area is approximated as a plane, and its normal direction is recognized as the three-dimensional posture of the part. do. That is, the area of the main plane surrounded by the jump edge and the roof edge is approximated by the least squares method. The plane equation to be found is t ₁ x + t ₂ y + t ₃ = z ...(6), and the xyz coordinates of m points in the main plane extracted from the range image are (ar1, ar2, bi) i = 1, 2, ...,
If m, t ₁ , t ₂ , t ₃ in equation (1) are (tj)j=1, 2,
Assuming a matrix t with 3 as a row, it is calculated as t=(A ^T A) ^-1 A ^T b (7). Here, A is a matrix whose rows are (ar1, ar2, 1.0), b is a row example whose rows are (bi), T is a transposed matrix, and ^-1 is an inverse matrix.

Note that the position of the center of gravity of the main plane area has already been determined during segmentation. The normal direction of this main plane is expressed by the angle between the normal and the Z axis, the angle θ between the projection of the normal on the x and y planes, and the X axis.

and θ are calculated using t (formula (6)) obtained during plane approximation, It is expressed as θ=tan ^-1 t ₂ /t ₁ ...(10).

Furthermore, as shown in Fig. 35, the parameters of the principal plane are rotated according to the value of θ, and x', y',
By converting to z', the shape of the main plane seen from the normal direction can be recognized.

In this way, the microprocessor 48 can determine the location and spatial tilt of the major surfaces of the object of interest relative to the robot's hand mechanism. Whether this is the surface of a connector can be determined from the size and shape of the surface as described above. Also, by detecting the direction in which the wire _A2 is located nearby, it can be determined that the opposite side is the direction in which the pin should be inserted. The remaining two sides are the surfaces that the robot finger should grip. Note that the ROM 49 of the image processor 21 has the following functions: jump edge detection and segmentation, separation and extraction of the target part, extraction of the plane area of the part by roof edge detection, and three-dimensional orientation of the part based on the normal direction of the plane. Programs for recognition, parts recognition, etc. are stored. RAM51b is the arithmetic unit 53
This is used to temporarily store the calculated data.
93 is a bus line.

The interface 52 of the image processor 21 sends data on the position, orientation, and direction of the main surface of the object to the robot controller 18.
That is, since the grasping point of the robot's fingers 25, the position to approach, and the direction thereof have been determined, the microphone processor 29 of the robot control device 18 determines these points.
In addition to the robot arm control data stored in the ROM 33, it is converted into a robot control coordinate system and transmitted to the drive circuit 31 via the D/A converter 30. Each actuator M provided in the robot 17 is driven, and the fingers of the robot 17 grip the connector 90 based on the above information and insert it into the pin 92 located at the position given in advance by teaching.

In the embodiment of the present invention, steady slit light is used as the light cutting method, but strobe light may be used instead. Further, in the embodiment of the present invention, a slit light is used, but it may be a bright and dark straight edge where one side is bright and the other side is dark. Further, instead of the slit light, a method may be adopted in which the light cutting surface 25 is scanned with a spot light.

Furthermore, in order to obtain a distance image from the light cutting detection head 77 attached to the hand mechanism 24 of the robot 17,
As in the embodiment shown in FIG. 29, in addition to linearly scanning by the detection head scanning mechanism provided at the tip of the hand mechanism 24, the robot controller 18 also controls the robot's hand mechanism 24 to move in the y-axis direction. It is clear that it is sufficient to send the coordinate values to the image processor 21 and sample them at constant intervals P while moving at a certain speed. However, robot hand mechanism 2
4 is moved in parallel to scan the 29th
The response is poorer than the embodiment shown in the figure.

In addition, in order to obtain a distance image from the light-cutting detection head, a detection head scanning mechanism is essential, and in addition to the linear movement mechanism shown in FIG. rotating shaft 94
Even if the mechanism is such that the imaging device and the slit projector 39 are oscillated by the rotating motor 95 around the third
As illustrated in FIG. 8, a mechanism for swinging only the slit projector 39 may be used.

In this case, the distance image is not proportional to each rotation angle and becomes a complex relational expression including the number of triangular strokes, making it extremely complicated to determine the inclination, direction, and position of the plane where the target object is located. However, when it is not necessary to accurately determine the inclination, direction, and position of the plane on which the object is located, the above relational expression can be replaced with an approximate expression, so
A swinging mechanism can be used as the scanning mechanism.
However, as a scanning mechanism attached to the hand mechanism 24, a swinging mechanism can be simpler as a mechanism than a linear scanning mechanism.

Furthermore, as described above, it is of course possible to use a method other than the light cutting method as a distance image detector.

FIG. 24 shows an embodiment of a detector using a diagonal cross slit beam 46 as an image detector attached to a robot arm. In this embodiment, two slit light sources 39a and 39b are used as diagonal cross-shaped slit light 4.
A TV camera 45 is used to detect the light cutting waveform when the two slit light sources 39a and 39b are emitted one by one.

A waveform signal is extracted from the TV image signal using the optical cutting line extraction device 42, and the actual distance to the object is determined using data 27 indicating the correspondence between the waveform signal value obtained in advance and the actual coordinates. Detect posture, etc. For details of this example, please refer to the patent application filed in 1983.
-Described in No. 201931.

An example of detecting the position and posture of an object using the robot vision device of the present invention will be described below.

Here, in order to make the explanation easier to understand, the apparatus configuration shown in FIG. 39 will be explained as an example.
Incidentally, the following explanation can be applied in exactly the same manner to the apparatus configuration shown in FIG. 40.

In FIG. 39, two slit light sources 101
When a and 101b are emitted one by one, the correspondence between the light cutting line detected by the imaging device 2, the actual position of the object, the equispaced line W, and the equidistant line D is shown in Fig. 41a. , b, it forms a diagonal grid. This correspondence relationship is determined in advance using an object whose dimensions are known while changing its position from the detector and stored in a storage device. When detecting the position and orientation of an object, the actual position and orientation of the object is determined by comparing this pre-stored correspondence with the position on the image of the light cutting line detected by the imaging device 102. You can find the position and tilt. In the present invention, two slit lights 103a, 1
Since the distance, position, and inclination to the object in two planes that are not parallel to each other are determined by switching the 03b and emitting light, the three-dimensional position and orientation of the object can be uniquely determined from these.

Note that in the present invention, two slit lights 103a,
It is assumed that the detector or the object is roughly aligned in advance so that the intersection line 106 of 103b intersects with the surface of the object, and the relative position of the detector and the object is determined using the robot vision device of the present invention. and posture shall be accurately detected.

Next, a method for determining the correspondence between the position of the detected light section line on the image and the actual position of the object will be explained using FIG. Here the third
Although only the case of the slit light source 101a shown in FIG. 9 will be explained, the slit light source 1 shown in FIG.
01b and the slit light source 101 shown in FIG.
The same applies to a and 101b. As shown in FIG. 42, the coordinate system xyz is such that the intersection line 105 of two slit beams coincides with the z-axis, and the slit beam 103a from the slit light source 101a coincides with the xy plane.
Assume that First, as shown in FIG. 42, a plane 106 perpendicular to the z-axis and wider than the field of view of the imaging device 102 is
is placed on a scaled rail 107 so that it can be moved in parallel in the z-axis direction. By moving the plane 106 away from the detector in steps of 1 cm, for example, and finding the position of the light cutting line on the detected image, an equidistant line diagram as shown in FIG. 43 is obtained. Next plane 10
6 is a rectangle with a constant width, such rectangles are prepared in several widths, and are also erected on the rail 107 perpendicular to the z-axis and moved in the z-axis direction. For example, the width measured in the x-axis direction of the plane 106 is set in 2 cm increments, and each plane 106 is
When the locus of the end point of the optical section line detected when the light section is moved in the axial direction is determined, a monospaced line diagram (regarding the x coordinate) as shown in FIG. 44 is obtained. In the equidistant line diagram shown in FIG. 43 and the equispaced line diagram shown in FIG. The result will be as shown in FIGS. 45 and 46. These can be expressed as distance functions z and x at vertical coordinate j on the diagram.
When the horizontal coordinate i on the diagram is expressed as a variable as a function x of the position in the direction, zj=f(i)=aj0−aj1i+aj2i ² +aj3i ³ +…
(12) xj=g(i)=bj0−bj1i+bj2i ² +bj3i ³ +…
It can be approximated as (13). The degree of approximation is determined by the required detection accuracy. For every j on the diagram (for example, j = 1 to 256),
If ajn, bjn (n = 0, 1, 2, ...) are determined in advance using the above method, then using equations (1) and (2) from the detected light cutting line, the z direction and You can find the position in the direction. In addition, when determining the inclination of a plane constituting the detection target, from the actual coordinates (x, z) (x', z') of the end points of the light section line determined as above, θ _x = tan ^-1 z'-z/x'-x (14) The angle between that plane and the x-axis can be found.

Next, the means for extracting the optical cutting line from the detected image will be specifically explained. In this case, the 39th
The case of the slit light beam 103a on one side shown in the figure will be explained. The optical cutting line imaged by the imaging device 102 generally has a certain width due to the width of the slit light, the blur of the optical system, etc. Therefore, the horizontal scanning direction of the imaging device 102 is arranged to match the direction (horizontal direction) of the line hh' shown in FIGS. 43 and 44, and the peak position (i.e., the brightest point) of each horizontal scanning signal is Let the coordinate i be the position i of the optical cutting line at each j (see FIG. 47). In this way, the coordinates of the position of the optical cutting line for calculating equations (12) and (13) are extracted.

Next, the process of detecting the three-dimensional position and orientation of an object using the robot vision device of the present invention will be described as follows: Detection of a cylindrical object 109 shown in FIG. 48 and a circular hole 110 on a plane shown in FIG. 149. This will be explained using an example.

First, a method for detecting the position and orientation of the cylindrical object 109 will be described. Suppose that when the slit light 103a shown in FIG. 40 is projected, for example, a light cut image as shown by the solid line A in FIG. 50 is detected by the imaging device 102. In FIG. 51, when the position of the brightest point in the horizontal direction is detected as described above and the shape of the light section line is extracted, it becomes as shown in FIG. When the waveform shown in FIG. 51 is approximated by a broken line and the coordinates (i, j) of the end points of each line segment are converted into actual coordinates (x, z) using equations (12) and (13) obtained in advance, we get The result will be as shown in FIG. In this figure, a cylindrical object 109 is traced from the midpoint of the line segment A with the smallest z-coordinate, that is, the line segment closest to the detector.
An approximate center position in the x direction of the upper surface of is determined. Furthermore, the inclination of the object 109 in the xz plane is determined using equation (14). Next, by switching the slit light source and performing the same detection on the optically sectioned image (indicated by the broken line B in FIG. 50) obtained for the other slit light 103b shown in FIG. Approximate center position of upper surface 109 in y direction and object 109 in yz plane
Find the slope of Furthermore, based on the above detection results, the distance z from the detector to the center position of the top surface of the object 109 is z=z ₁ + z ₂ when the top surface of the cylindrical object 109 is approximately perpendicular to the z-axis. /2+z ₄ −z ₃ /y ₂ −y ₁・y ₁ +y ₂ /2(
15) Or, z= _z3 + _z4 /2+ _z2 - _z1 / _x2 - _x1・_x1 + _x2 /2(
16). Here, x ₁ , x ₂ are the x coordinates of the end points of the optically sectioned image (A shown in FIG. 50) of the xz plane on the upper surface of the cylindrical object 109, and y ₁ , y ₂
is the y-coordinate of the end point of the optically sectioned image on the yz plane (B shown in Figure 50), and z ₁ , z ₂ , z ₃ , and z ₄ are each 4
represents the z-coordinate of the two endpoints. From the above detection results, if the xz plane and yz plane of the top surface of the cylindrical object 109
If the inclination in the plane is significantly different from 90° and it is determined that the method of determining the center position from the midpoint of the light section line and the error in calculating the distance using equation (15) or (16) cannot be ignored, the inclination The robot arm is moved according to the detection result so that the z-axis, that is, the line of intersection of the slit beams 3a and 3b, is substantially perpendicular to the plane, and the center position and distance are detected again.

Next, a method for detecting the center position and direction of the circular hole 110 perpendicular to the plane shown in FIG. 49 will be described. FIG. 53 is a diagram showing an example of a light section line obtained when the hole 110 shown in FIG. 49 is imaged by an imaging device. In this case, as in the previous example, two sets of optical cutting lines in the xz plane and the yz plane are detected independently by switching the slit light source. The solid line and broken line in FIG. 53 indicate two sets of optically sectioned images obtained in this way. In the same figure, by processing the line segment connecting points a ₁ and a ₂ and the line segment connecting points b ₁ and b ₂ to correspond to the line segments A and B in FIG. 50 of the previous example, respectively, The position of the hole 110 and the direction in which the hole is formed can be detected in exactly the same manner.

Below, specific embodiments of the robot vision device of the present invention will be described. First, the overall structure and operation of the apparatus will be explained, and then the specific structure of each part of the detector will be described.

FIG. 24 is a diagram showing an embodiment of the overall configuration of a robot visual system according to the present invention. The detector includes an imaging device 2 and slit light sources 39a, 101a, 39b, 1
01b, and is arranged and constructed as follows. That is, (1) two slit lights 46a, 103a, 46
The intersection line 105 of the planes formed by b and 103b is made to coincide with the rotation axis of the gripper 25 of the robot.

(2) Optical axis 108 of slit light sources 39a, 39b
a, 108b and the optical axis 108c of the imaging devices 45, 102 intersect at approximately one point on the intersection line 105 of the slit lights 46a, 46b.

(3) Optical axis 108 of slit light sources 39a and 39b
a, 108b and the optical axis 108c of the imaging devices 45, 102 are on the same plane.

(4) The angle between the optical axis 108a of the slit light source 39a and the intersection line 105 of the slit light and the slit light source 3
Intersection line 105 of optical axis 108b of 9b and slit light
is equal to the angle formed by

(5) The angle between the optical axis 108a of the slit light source 39a and the optical axis 108c of the imaging devices 45 and 102, and the angle between the optical axis 108b of the slit light source 39b and the imaging device 4.
The angle formed by the optical axis 108c of No. 5 is equal to the angle formed by the optical axis 108c of No. 5.

A detector is configured on the robot arm 24 with the arrangement as described above. Other modified examples of the detector configuration method include logically possible 1 to 4 of the above (1) to (5).
There are several combinations. Further, in FIG. 24, the detector is fixed to a portion of the robot that rotates the gripper 25, but it may be fixed on the gripper 25 and rotated integrally with the gripper 25.

The entire robot visual system is controlled by an image processor 47 shown in FIG. That is, the slit light switching device 113 is controlled so that the slit light sources 39a and 39b emit light one by one, and the respective light section images are captured by the imaging devices 45 and 10.
Detected by 2. The detected image signal is input to the optical cutting line extraction circuit 42, and the optical cutting waveform is input to the image processor 47. The optical cutting line extraction circuit 42 compresses two-dimensional image data into one-dimensional waveform data according to the method described above, and may use the device disclosed in Japanese Patent Application Laid-open No. 70407/1983. . The two sets of optical section line waveform data thus obtained are processed in the image processor (microcomputer) 47 according to the procedure described above. In other words, after the polygonal line approximation of the waveform data, calculations of equations (12) to (16) are performed to calculate the three
Detect the dimensional position and orientation and collect the data 117
is output to the control device of the robot arm mechanism. Coefficients ajn, bjn necessary to calculate equations (12) and (13)
(j=0, 1, 2, 3..., n=0, 1, 2, 3...)
is determined in advance using the method described above, is stored in the storage device 116, and is read out and used as needed.

Next, the operation of the robot vision device of this embodiment will be explained in detail.

Before detecting the object, as already explained, it is necessary to store the coefficients ajn and bjn (and the coefficient of yj obtained in the same way) shown in equations (1) and (2) in the storage device 116. It is. Using the device shown in FIG. 42, the robot is operated and fixed so that the intersection line 105 of the slit beams is parallel to the rail 107, and the equidistant line diagram shown in FIG.
Obtain the monospaced line diagram shown in Figure 4. This operation is performed by switching between the two slit lights 4 using the slit switching device 113.
6a, 103a, 46b, and 103b, respectively. Next, based on these data, the image processor 47 performs polynomial approximation,
The coefficients ajn and bjn of equations (12) and (13) for each j are determined and stored in the storage device 116.

In the target object detection process, as already explained using FIGS. 48 and 49 as examples, the image processor 47 controls the slit light switching device 113 to cause the two sets of slit light sources 101a and 101b to emit light, respectively. A photocutting waveform based on the photocutting image in the case of It is calculated using equations (12) and (13) (and similar equations for finding yj) based on the coefficients ajn and bjn stored in . The image processor 47 selects one line segment on the target object from among the line segments when the two sets of slit light sources 101a and 101b are emitted, using the distance z as a criterion (usually the most detected line segment). (line segment near the vessel and near the center of the field of view), from each of its four endpoints, Equation (14)
- (16) and calculate the midpoint of the line segment to detect the position and orientation of the target object and output it to the control device of the robot arm mechanism.

Next, the specific configuration of each part of the detector will be explained. A TV camera is used as the imaging device 45, and the 47th
By matching the direction of i shown in the figure with the horizontal scanning direction, the optical cutting line can be detected at high speed. Of course, a two-dimensional image detector other than a TV camera, such as a combination of a one-dimensional photodiode array and a galvanometer mirror, may also be used. Among TV cameras, those that use two-dimensional solid-state image sensors in particular have the following characteristics: (a) earthquake resistance and shock resistance, (b) lifespan, (c) image distortion, (d) small size and light weight, and (e) power saving. It is superior to those using electron tubes in five respects. FIG. 54 is a diagram showing a specific example of an imaging device using a two-dimensional solid-state image sensor 119. In this specific example, the part mounted on the robot arm is the TV camera control unit 1, as shown by the dotted line in FIG.
22, there are only the two-dimensional solid-state image sensor 119, the front image signal amplifier 120, and the synchronous drive signal waveform shaping circuit 121, and further reduction in size and weight is achieved. In FIG. 54, a front image signal amplifier 120 is mounted on a substrate 123a, a two-dimensional solid-state image sensor 119 and a synchronous drive signal waveform shaping circuit 121 are mounted on a substrate 123b, and are fixed on a back cover 125 with a support 124. There is. A connector 126 is attached to the back cover 125, and all signals and power are supplied via this connector 126. In addition, the back cover 125 and the outer frame 12
7 is made of a non-metallic material such as plastic with copper foil for electrostatic shielding applied to the inner surface (or metal plating applied). Furthermore, the imaging lens 128 can also be made of plastic. As described above, if the imaging device according to this embodiment is used, earthquake resistance and
In addition to features such as impact resistance, long life, low image distortion, and low power consumption, it also has the effect of significantly reducing the size and weight of the detector.

Next, the specific configuration of the slit light sources 101a and 101b will be explained. As a light source, halogen lamps, discharge tubes, light emitting diodes, etc. can be used, but they have the following characteristics: (a) seismic/impact resistance, (b) lifespan, and (c)
Light-emitting diodes are the best in three respects: small size and light weight. FIG. 56 is a diagram showing a specific example of a slit light source using a light emitting diode. The light emitting diode 129 is of a high output type that emits infrared light. Since the maximum sensitivity wavelength of a two-dimensional solid-state image sensor is usually around 800 nm, detection sensitivity can be maximized by using a light emitting diode of this wavelength. As shown in FIG. 56, a light emitting diode 129
For example, a plurality of lenses are arranged in parallel in the longitudinal direction of the cylindrical lens 130. 56th
In the plan view shown in Figure a, the light emitting diode 12
9 and slit 131 are cylindrical lenses 13
There is an imaging relationship with respect to 0. In addition, the imaging plane 132
The position of is set to the approximate object position, and the imaging plane 13
2 and the slit 131 are in an imaging relationship with respect to the imaging lens 133. By adopting the above configuration, the light emitted from the light emitting diode 129 can be efficiently collected and a slit beam can be formed.
In addition to the above three features, a bright slit light source can be realized. Furthermore, by using a lens-based light emitting diode 29 with strong directivity, an even more powerful slit light source can be realized.

FIG. 57 is a diagram showing another specific example of a slit light source using a light emitting diode. As shown in FIG.
31, and the slit 131 and the imaging surface 1
32 is in an imaging relationship with respect to the imaging lens 133. Further, the cylindrical lens 130 is inserted between the imaging lens 133 and the slit 131 so that the slit light does not become uneven on the imaging surface 132. In this specific example, if the imaging relationship of the imaging surface 132 is out of order, the slit light may become uneven.
This can be ignored in the present invention, which detects the position of the optical cutting line. According to this specific example, as is clear from a comparison with FIG. 22, the length in the optical axis direction can be shortened, and in addition to the characteristics of the light emitting diode 129, the slit light source can be made significantly smaller. There is.

Further, like the imaging device, if the outer frame is made of non-metal such as plastic, the detector can be made significantly smaller and lighter.

Note that in the specific examples of the slit light source shown in FIGS. 56 and 57, the slit light does not have a perfect flat shape. This problem can be avoided by not making the diameter of the imaging lens 133 larger than necessary.

Other embodiments of the image detector 20 mounted on the robot arm include a grayscale image detector using a combination of a TV camera, a linear sensor, or a point sensor (photodiode, photomal, etc.) and a scanning mechanism such as a galvanometer mirror. In the present invention, before the recognition processing by the image detector on the robot arm, the image from the distance image detector at a fixed position is processed, and the robot arm approaches the target object from the normal direction of the main plane. Since the image can be detected from that direction by the image detector on the robot arm, it is possible to detect the correct planar shape of the main plane. position,
Can recognize posture. In addition, if the background and the target object can be detected separately by binarizing the grayscale image,
Using value image processing, such as template matching, the shape, position, and orientation of a target object can be detected at high speed.

The position and orientation of the target object obtained by processing the images from the image detector on the robot arm described above are based on the coordinate system fixed to the image detector, that is, the relative positional relationship with the robot arm. Therefore, even if the absolute positioning accuracy is poor due to deflection of the robot arm, the robot arm can be made to approach the target object with high precision. Further, in this case, the robot control device needs to calculate the amount of movement of each axis of the robot from the posture of the robot arm at the time of detection from the robot arm and the position and posture of the target object from the image processor.

Next, FIG. 25 shows an embodiment of the image processor 21, which is one of the other components of the visual system.
The image processor 21 includes a distance image detector 19
The image detected by the image detector 20 on the robot arm is processed in accordance with the method described above, the target object 11 is discovered, its position and orientation are recognized, and the information is transmitted to the robot controller 18. The image processor 21 is a microprocessor 48
It is controlled by the ROM 49 and operates according to a program stored in the ROM 49. The optical cutting line extraction device 42 for generating a distance image and the image detector 20 on the robot arm are a high-speed parallel I/F 50a,
The image information connected and detected by 50b is
It is stored in the RAM 51a. In addition to being used to temporarily store image information in this way, the RAM 51 is also used as a work area in the image processing process. It is also connected to the robot control device 18 via an interface 52 to transmit recognition results.

The embodiment of the image processor 21 described above is as follows.
Although we have described the case where software processing is performed using the microprocessor 48, for example, binarization processing, differentiation processing, etc. can be executed with a relatively simple hardware configuration, so it is possible to partially perform processing by hardware. If you do this, you can speed up the processing. In addition, since extremely high-speed processing is performed, it is possible to implement all processing algorithms in hardware, although the scale of the device becomes larger.

Further, in the embodiments of the present invention described above, an articulated robot has been described, but it is clear that an orthogonal robot or a cylindrical robot may also be used.

Furthermore, as another modification of the present invention, depending on the work object, the target object may be recognized only by the distance image detector 19, or the image detector 2 on the robot arm may be used.
It should be added that it is also possible to recognize a target object using only 0.

〔Effect of the invention〕

As explained above, according to the present invention, it is possible to realize robot control using complete visual feedback, and furthermore, it is possible to realize the robot control using complete visual feedback, and also to detect the workpiece based on the base part on which the robot is fixed using the distance image detection means, and to detect its posture and position. This step-by-step process of correcting the position and orientation of the robot based on the recognition of the position and orientation of the workpiece based on the robot image detector allows the robot to This has the effect that a work target can be found and a robot can perform work in a wide range of fields such as high-precision assembly, inspection, and adjustment on the work target without performing extremely complicated search operations. Further, according to the present invention, since the distance image detection means detects the object to be worked on and recognizes its posture and position with reference to the base on which the robot is fixed, the color, brightness, and surface of the surface of the object to be worked on are recognized. Efficient recognition is always possible without being affected by roughness, dirt, etc. Furthermore, distance image signals can be divided into two
Since the boundaries of the workpiece are detected through dimensional differentiation, the workpiece can be found stably regardless of the surface condition of the workpiece.

[Brief explanation of drawings]

Figures 1 and 2 are diagrams showing conventional robot vision, Figure 3 is a diagram explaining a distance image, Figure 4,
5, 6, 7 and 8 are diagrams showing the process of processing a distance image according to the present invention, FIG. 9 is a diagram showing two ways of viewing a rectangular parallelepiped, and FIGS.
1 is a diagram showing the roof edge and jump edge of a distance image, FIG. 12 is a diagram showing the definition of the posture of an object by distance image processing, and FIG. 13 is a diagram showing the overall configuration of an embodiment according to the present invention. Fig. 14 is a diagram explaining the robot control device, Fig. 15 is a diagram showing an embodiment of the distance image detector, Fig. 16, Fig. 1
Figures 7, 18, 19, and 20 are diagrams showing the detection process, Figure 21 is an example of another distance image detector, and Figure 22 explains the cause of shadows in distance images. 23 and 24 are diagrams showing an embodiment of the image detector on the robot arm, and FIG.
The figure shows an example of an image processor.
FIG. 26 is a schematic configuration diagram showing the optical cutting line extraction circuit according to the present invention, FIG. 27 is a diagram for explaining the function of the circuit shown in FIG. 26, and FIG. 28 is a diagram showing the circuit shown in FIG. 27 in detail. The configuration diagram shown in Figure 29 is the second
3 is a diagram specifically showing an embodiment of the robot visual device according to the present invention, FIG. 30 is a diagram for explaining the work object in the embodiment shown in FIGS. 23 and 29, and FIG. The figure shows an example of a light section line, FIG. 32 is a diagram for explaining the light section extraction process, FIG. 33 shows an example of a distance image, and FIG. 34 shows the light section line and the center of the imaging device. FIG. 35 is a diagram showing the jump edge and roof edge in a distance image, and FIG. 36 is a diagram showing the relationship between the distance image signal and the target object surface. FIG. 37 is a diagram showing a state in which the distances are cut out, and FIG.
FIG. 38 is a diagram showing another embodiment different from the figure, and FIG. 38 is a diagram showing another embodiment according to the present invention.
9 is a perspective view of a detector which is the principle of the robot cross cutting line vision device according to the present invention as shown in FIG. 24, FIG. 40 is a perspective view showing the basic configuration of the detector shown in FIG. 39, and FIG. The figure shows equidistant lines and equispaced lines obtained by the detector shown in Fig. 39, Fig. 42 is a perspective view of the detector for obtaining equidistant lines and equispaced lines, and Fig. 4
Fig. 3 is a diagram showing an example of equidistant lines, Fig. 44 is a diagram showing an example of equidistant lines, Fig. 45 is a diagram showing changes in the equidistant lines shown in Fig. 43, and Fig. 46 is a diagram showing an example of equidistant lines. 47 is a diagram showing the method of extracting the light cutting line, and 48 and 4
Figure 9 is a diagram showing an example of an object to be detected; Figure 50;
51 and 52 are diagrams showing the process of detecting the object to be detected shown in FIG. 48, FIG. 53 is a diagram showing the detected image of the object to be detected shown in FIG. 49, and FIG.
55 is a diagram showing an example of the overall configuration of the imaging device shown in FIG. 24, FIG. 55 is a diagram showing a signal processing circuit of the image sensor shown in FIG. 54, and FIGS. 56a and 57a are A plan view showing a specific example of the slit light source used in the detector of the robot vision device according to the present invention, FIG. 56b and FIG. 57b show a specific example of the slit light source used in the detector of the robot vision device of the present invention. It is a front view. 9...Distance image, 11...Target object, 17...
Robot arm, 18... Robot control device, 1
9... Distance image detector, 20... Image detector on robot arm, 21... Image processor,
38...Slit light, 39...Slit light source, 4
2... Optical cutting line extraction device, 45... TV camera.

Claims (1)

[Scope of Claims] 1. A robot having a movable member, and information on the distance from a point determined with respect to a base portion to which the robot is fixed to each point in the entire working area of the robot as a two-dimensional distance. A distance image detection means that detects the image signals all at once; a distance image signal detected by the distance image detection means is two-dimensionally differentiated; and based on the two-dimensionally differentiated two-dimensional differential signal,
The work target is discovered by extracting a two-dimensionally surrounding area, and the normal direction and position of the surface of the work target are determined based on the distance image signal detected by the distance image detection means in the extracted area. a first image processing means for determining the normal to the surface of the workpiece determined by the image processing means; a robot image detector fixed to a movable member of the robot; robot control means for controlling the position and attitude of the robot by driving and controlling the movable parts of the robot so as to have a specific angle and distance relationship with respect to the direction and position; a second image processing means for calculating the position and orientation of the work object from the image signal with reference to the robot image detector; and a work object calculated from the second image processing means with reference to the robot image detector. A robot control device comprising: correction control means for correcting the position and orientation of the robot by driving and controlling the movable parts of the robot based on the position and orientation of the object. 2. The image processor includes a differentiating means for differentiating a distance image signal to form a jump edge image signal, a separating means for separating into regions closed by the jump edge, and a separating means for extracting a necessary region from the separated regions. , is characterized by comprising an extraction means for extracting a distance image of the main plane from this extracted area, and a detection means for detecting the position and inclination of the center of gravity of this main plane from this distance image. The robot according to claim 1.