JPH079673B2 - Corresponding point determination device for binocular vision system - Google Patents

Description

Description: TECHNICAL FIELD OF THE INVENTION The present invention relates to a three-dimensional object recognition device applied to industrial robots and the like, and particularly, the present invention relates to a two-visual system.
The present invention relates to a corresponding point determination device for a binocular vision system that determines the corresponding points by associating the characteristic points of the images respectively input from a single imaging device.

<Background of the Invention> Usually, in order to optically stereoscopically recognize a three-dimensional object, it is necessary to use a pair of two visual systems. When an image of an object is formed on this binocular vision system, a gap between the images occurs due to the separation of the visual system. By detecting this shift from the corresponding points of both images, the object is stereoscopically grasped. Is possible. Therefore, in this case, it is necessary to make the points of the image related to the one visual system correspond to the points of the image related to the other visual system.
Japanese Unexamined Patent Publication No. 58-90268 discloses a first binocular stereoscopic system.
An apparatus for determining corresponding points between images obtained with each second camera is described. First of all, after deciding a pattern to be noticed from the image obtained by the first camera and setting the pattern at the center of the visual field, the line-of-sight vector of the first camera forms a straight line on the image obtained by the second camera (this is called "epipolar Line)), and a pattern having a shape similar to the pattern of interest on the epipolar line is searched for as a candidate for the corresponding pattern. Next, after placing this candidate at the center of the field of view of the second camera, the three-dimensional coordinates of the object are calculated from the intersection of the central axes of the first and second cameras, and the 3
Calculate the distance from each camera to the object in the dimensional space.
Next, the degree of coincidence between the corresponding patterns obtained by the first and second cameras is calculated. If the degree of coincidence is less than the threshold value, it is determined that the two patterns do not correspond to each other, and the pattern is equal to or more than the threshold value. If, then the confirmation operation is performed. In this confirmation operation, the entire binocular stereoscopic system is moved by a certain amount in the line-of-sight direction of the object seen from the first camera, and the line-of-sight of the second camera is directed toward the previously obtained intersection point to determine the first line of sight. Then, the degree of coincidence between the patterns by the second camera is calculated, and the distance from each camera to the object is calculated. If the matching degree is equal to or greater than the threshold and the condition that the distance from the camera to the object changes by the moving distance before and after the movement is satisfied, both patterns are To determine. However, this device requires pattern comparison processing and three-dimensional coordinate measurement processing in order to associate the patterns by the first and second cameras.
There is a problem in that the amount of calculation increases remarkably, it takes a long time for the association, and the processing becomes complicated.

<Objects of the Invention> The present invention has been made in view of the above problems, and is directed to a binocular vision system capable of determining corresponding points between images at high speed without performing pattern comparison processing or three-dimensional coordinate measurement processing. An object is to provide a corresponding point determination device.

<Structure and Effect of the Invention> In order to achieve the above object, according to the present invention, a binocular vision unit including two imaging devices capable of observing the same object point from different positions, and the binocular vision The displacement means capable of displacing the means, and the feature points of the images respectively inputted from the two image pickup devices constituting the binocular vision means,
It is said that the coordinates of the feature points of the image obtained by one of the image pickup devices and the coordinates of the feature points of the image obtained by the other image pickup device are located on the same horizontal line that is established between the feature points of the same object point. Using the constraint condition of No. 1, the feature points are associated with each other and the corresponding point set candidate extracting means for extracting the corresponding point set candidates, and the image obtained by the binocular vision means after being displaced by the displacing means The second feature is that they are located on a transitional horizontal line that is established between the feature points of the same object point between the coordinates of the feature points and the coordinates of the feature points of the image obtained by the binocular vision means before displacement.
The corresponding point set determining means for narrowing down the corresponding point set candidates extracted by the corresponding point set candidate extracting means and determining the corresponding point set by using the constraint condition Decided to configure.

According to the present invention, with respect to the feature points of the image before and after the displacement obtained in the binocular vision system, the corresponding point set candidate is extracted by the processing only in the coordinate system of the image using the first and second constraint conditions. Since the corresponding point set is determined, the amount of calculation is greatly reduced and the corresponding points between the images are determined at high speed as compared with the conventional example that requires pattern comparison processing and three-dimensional coordinate measurement processing. The effect is that you can do it.

<Description of Embodiments> The object recognition device shown in FIG. 1 has a pair of left and right visual systems 1L and 1R whose optical axes are parallel to each other. Each of the visual systems 1L and 1R is composed of a TV camera of the same magnification.
The position and orientation of 1L and 1R can be changed with high accuracy. The object image formed on each of the visual systems 1L and 1R is taken into an image memory (not shown) of the image processing device 3. The image processing device 3 uses the two object images to characterize points (hereinafter,
Feature points) are extracted, and the object is stereoscopically recognized based on the shift between corresponding feature points. The illustrated device is
It has a function to specify and extract a set of corresponding feature points of the object image for each visual system 1L, 1R.

2 and 3 specifically show the principle of extracting such feature points.

The xyZ coordinates shown in FIG. 2 indicate a positional relationship between a pair of left and right visual systems, a point on the object (hereinafter referred to as an object point), and an image of the object point (hereinafter referred to as an image point). Further, in the case of the illustrated example, the image point imaged in the negative region of the Z-axis is projected to the position of Z = 1 in the positive region for ease of explanation. In the figure, B is the object point, L
And R are the optical axes of the left and right visual systems, and P _L and P _R are the image points of the left and right visual systems. Also, the intersection of the left and right optical axes L, R and the y-axis is the image of the intersection of the perspective centers S _L , S _R , and the intersection of the left and right optical axes L, R and the left and right image planes T _L , T _R set at Z = 1. The center points are O _L and O _R.

The coordinates of the object point B are (X, Y, Z), and the perspective centers S _L and S _R are (0, −
a, 0) (0, a, 0), and the image points P _L and P _R are (x _L , y _L , 1) (x _R , y _R ,
If it is 1), the following equations (1) to (4) are established.

Next, the left and right visual systems 1L and 1R are displaced in parallel with the displacement α in the x-axis direction, and the image points when rotated by the angle θ around the Z axis after displacement are P _L ′ and P _R ′, and the coordinates are (X _L ′, y _L ′, 1)
Assuming that (x _R ′, y _R ′, 1), x _L ′, y _L ′, x _R ′, y _R ′ are represented by the following equations (5) to (8).

Thus (1) and (3), from (5) and _{(7), x R = x L ......... (} 9) x R '= x L' ......... (10) established To do. In the equations (9) and (10), the image points P _L and P _R of the left and right visual systems for the same object point are located on the same horizontal line, and the left and right visual systems are translated and rotated. The latter image points P _L ′ and P _R ′ are also located on the same horizontal line.

The (9) and (10), (2) (4) (6) from the equation _{(8), x R '-x} R = x L' -x L ......... (11) y R ' −y _R ＝ y _L ′ −y _L ………… (12) holds. The equations (11) and (12) represent the image points of the left and right visual systems when the left and right visual systems are translated and rotated.
It shows that P _L and P _R are displaced by the same amount in the same direction.

Therefore, if the displacements α and θ of both visual systems are determined, the image point P _L ,
For the distances between P _L ′ and between the image points P _R and P _R ′, the following equation holds.

However, TH ₁ is a threshold determined by the displacements α and θ.

FIG. 3 is a diagram showing the positional relationship between the object image before displacement and the object image after displacement for the left and right visual systems. In the figure, P _{1 to} P
₁₀ is a feature point extracted from the object image before displacement, P _{11 to} P ₂₀
Indicates the feature points extracted from the object image after displacement. Focusing now on the characteristic point P ₁ of the left visual system, the characteristic point P ₁ is located on the horizontal line h. Therefore, judging from the principle of the equation (9), the characteristic point of the right visual system corresponding to the characteristic point P ₁ is either the characteristic points P ₆ or P ₉ located on the same horizontal line h. Next, when both visual systems are displaced by a certain amount, the characteristic point P ₁ moves to the characteristic point P ₁₁ . However, only the feature point P _{16 associated with} the transition of the feature point P ₆ is located on the horizontal line h ′ passing through the feature point P ₁₁ , and the feature point P _{19 associated with} the transition of the feature point P ₉ does not exist. Therefore, from the principle according to the equation (10), it can be determined that the feature point P ₁ of the left visual system corresponds to the feature point P ₆ of the right visual system.

FIG. 4 shows an algorithm for object recognition including a process of identifying and extracting a set of corresponding feature points of the left and right visual systems.

First, at step 11, the object image formed on the left and right visual systems is loaded into the image memory of the image processing apparatus 3 and set. Next, in step 12, the left and right visual systems are displaced in parallel in the x-axis direction by α, and further, in step 13, the angle θ is rotated around the Z axis. Then, in step 14, the image of the same object is changed to the left and right visual systems. An image is formed, and similarly, it is taken into the image memory and set. At this stage, the image memory stores two object images before displacement and two object images after displacement. Next, in step 15, line drawing processing is executed for each of the four object images in total, and points where the contour line or the like changes sharply are extracted as feature points, and the coordinates of each feature point are calculated for each image. Then, the images are sequentially set in a predetermined area of the image memory. FIG. 6 shows an entire list in which all the characteristic points of the left visual system before change are set.

Now at step 16, the feature points labeled j in FIG.
Focusing on P _L (j), the coordinates of the characteristic point P _L (j) {x
_L (j), y _L (j)}. And the next step 17
Then, an extraction list of feature points of the right visual system located on the same horizontal line as the feature point P _L (j) (hereinafter referred to as horizontal point list) is created. This horizontal point list is indicated by HOR in FIG. 5, and the coordinates {x _R (s), y included in the horizontal point list HOR are shown.
The characteristic point P _R (s) of _R (s)} satisfies the following formula (15) from the principle of the above formula (9). Note that ε ₁ is a positive system constant close to zero.

│x _L (j) −x _R (s) │ ≦ ε ₀ (15) where s is a label in the whole list, and s = HO in the figure.
The expression R (i ₁ ) means that the feature point P _R (s) of the label s is given the label i ₁ in the horizontal point list HOR.

Next, at step 18, the coordinates {x _L ′ (q), y _L ′ (q which satisfy the following equation (16) (based on the equation (13)) are set for the feature points of the left visual system after displacement. )} Feature points P _L ′ (q) are extracted to create a neighbor point list NEIG.

However, q is a label in the whole list, and q = in FIG.
The expression NEIG (i ₂ ) indicates that the feature point P _L ′ (q) with the label q is given the label i ₂ in the neighbor point list NEIG.

Further, in the next step 19, a horizontal point list of characteristic points of the right visual system located on the same horizontal line as each characteristic point included in the neighboring point list NEIG is created. This horizontal point list is
In FIG. 5, the feature point P _R ′ indicated by RTAB and having coordinates {x _R ′ (u), y _R ′ (u)} included in the horizontal point list RTAB is shown.
(U) is the following (17) from the principle of the above equation (10).
Satisfy the formula.

│x _L ′ (q) −x _R ′ (u) │ ≦ ε ₀ ……… (17) where u is the label in the whole list, and u = RTAB in the figure.
The expression (i ₂ , i ₃ ) means that the feature point P _R ′ (u) of the label u is given the label i ₃ in the horizontal point list RTAB.

After completion of creation of each of the above lists, in step 20, corresponding feature points of both visual systems satisfying the following equation (18) based on the equation (12) and the following equation (19) based on the equation (14). Set {P _L (j), P _R (s), P _L ′ (q), P _R ′
(U)} is extracted. However, ε ₁ is a positive system constant close to zero.

| Y _L ′ (q) −y _L (j) −y _R ′ (u) + y _R (s) | ≦
ε ₁ ……… (18) If there is a valid set of feature points, the determination at step 21 is "YES", and at the next step 22, from the feature points P _L (j) P _R (s) before displacement, Next (20) (21)
According to the equation (22), the three-dimensional coordinates of the object point B (X _LR , Y _LR , Z
_LR ) is calculated.

Further, in the next step 23, the feature point P _L ′ (q) P _R ′ after displacement is
From (u), the three-dimensional coordinates (X _LR ′, Y _LR ′, Z _LR ′) of the object point B are calculated by the following equations (23), (24) and (25).

Next, the three-dimensional coordinates (X _LR , Y _LR , Z _LR ) (X _LR ′, Y _LR ′, Z _LR ′) of the object point calculated separately from the feature points before and after displacement
Then, the inter-coordinate distance DIST is calculated using the following equation (26).

If the inter-coordinate distance DIST is equal to or less than the constant threshold value TH ₂ , the determination in step 24 becomes “YES”, and in the next step 25, the data regarding the three-dimensional coordinate of the object point is set to the seventh value.
As shown in the figure, it is registered in the image memory according to the following equations (27) to (33).

B _L (k) = j ……… (30) B _R (k) = s ……… (31) A _L (k) = q ……… (32) A _R (k) = u ……… ( 33) If the above processes are repeated for all the feature points of the left visual system before displacement in the same manner, the determination in step 26 becomes "YES", and the coordinate calculation process of the object points including the feature point extraction process is performed. All done.

[Brief description of drawings]

FIG. 1 is a perspective view showing the schematic configuration of the object recognition device, FIG. 2 is a diagram showing the positional relationship among the left and right visual systems, object points, and image points in xyZ coordinates, and FIG. 3 is the corresponding feature points. FIG. 4 is a flow chart showing an algorithm of object recognition, FIG. 5 is a view showing a relationship between feature points and a feature list, and FIG. 6 is a left visual system before displacement. FIG. 7 is a diagram showing an overall list of feature points, and FIG. 7 is a diagram showing a registration state of data regarding coordinates of object points. 1L, 1R ... Visual system, 2 ... Driving device 3 ... Image processing device, B ... Object point P _R , P _L ... Image point

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Noriyuki Tsukiyama, No. 10 Hanazono Dodo-cho, Ukyo-ku, Kyoto City, Kyoto Prefecture Tateishi Electric Co., Ltd. (56) Reference JP-A-58-90268 (JP, A) 56-170707 (JP, U) JP-B 54-23545 (JP, B1)

Claims (1)

[Claims] 1. A binocular vision means comprising two image pickup devices capable of observing the same object point from different positions; a displacement means capable of displacing the binocular vision means; Regarding the feature points of the images respectively input from the two image pickup devices constituting the binocular vision means, the coordinates of the feature points of the image obtained by one of the image pickup devices and the coordinates of the feature points of the image obtained by the other image pickup device. Corresponding point set candidates for which the corresponding feature points are associated and the corresponding point set candidates are extracted by using the first constraint condition that they are located on the same horizontal line that is established between the feature points related to the same object point The same object point between the extraction means and the coordinates of the feature points of the image obtained by the binocular vision means after displacement by the displacement means and the coordinates of the feature points of the image obtained by the binocular vision means before displacement. When it is located on the transitional horizon established between the feature points A binocular vision system comprising: corresponding point set determining means for narrowing down the corresponding point set candidates extracted by the corresponding point set candidate extracting means by using the second constraint condition. Corresponding point determination device.