JPH0762871B2 - Image processing method - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a method and apparatus for processing image information converted into digital data on a pixel-by-pixel basis. More specifically, the same object to be processed can be processed at different positions. The present invention relates to a method and apparatus for identifying and measuring the position of an object in a three-dimensional space by processing a captured image.

[Prior art]

In order to process and measure the position of an object in a three-dimensional space by processing a plurality of images taken at different positions, it is necessary to specify the position of the image in each image of the taken object.

To this end, each point in the image, that is, the feature of each pixel that is the smallest unit that constitutes the image is identified, and the features of each pixel are compared and contrasted among a plurality of images to belong to the same object. It is necessary to perform a process of associating by specifying a combination of pixels that image the same point. Then, after that, the distance between the camera and the point is obtained from the coordinates of each pixel on the image based on the principle of triangulation. Further, the same processing is performed on all the pixels in which the object is photographed, and the distance and the shape of the object from the camera are obtained.

Conventionally, such processing is a method shown in “Correspondence based on contour line segment of stereo image” (Information Processing Society of Japan, Computer Vision Research Group material 38-1, 1985). This is a method of dividing an area into areas by edges and correlating a plurality of images with respect to a contour line surrounding the area.

For convenience of description, the simplest method using two cameras, that is, the case of processing a stereo image obtained by a stereo camera will be described here. Specifically, the second
Assuming that two images as shown in FIGS. 7A and 7B are taken by the left and right cameras, the contour line segment is extracted from each of the images. For example, taking the image on the right side of FIG. 2 (b) as an example, as shown in FIG. 3 (a), the edge strength and edge direction of each point of the image are obtained by a first-order differential operator, and the values are used to make a thin line. Is performed to obtain pixels having a certain threshold value or more. Further, the end points of each point sequence are obtained so as to form a closed area as much as possible, and extension processing is performed from that point in the direction orthogonal to the edge direction. The results are shown in Fig. 3 (b).

Next, the area surrounded by the edge points is labeled, and the area of each area, the average brightness, the spread in the vertical and horizontal directions, and the like are obtained.

The edge points surrounding each area are traced, and are divided into branch points and bending points as shown in the flowchart of FIG.
Fit the next curve. If the fitting error is large, it is subdivided at the point farthest from the straight line connecting the two end points, and the fitting is repeated for each of the divided point sequences. A straight line or a quadratic curve obtained by this method is used as a contour line segment.

Images of contour line segments obtained by processing FIG. 2 by this method are shown in FIGS. 5 (a) and 5 (b). Note that (a) and (b) in FIG. 5 correspond to (a) and (b) in FIG. 2, respectively.

By the way, as shown in Fig. 6 (a), the left and right TV cameras are
When installed such that the central axes of the fields of view are parallel to each other and perpendicular to a straight line connecting two cameras, an arbitrary point P ^* in the three-dimensional space is a point on the image captured by the left and right cameras. It maps to P (x _P , y _P ) and P '(x _P ', y _P '). The following equation (1) is established as a geometric constraint condition for the coordinates of the point P and the point P ′, which can be used as a limiting condition for performing correspondence between the left and right images. Hereinafter, this condition will be referred to as “epipolar condition” for convenience.

However, in order to increase the measurement accuracy sufficiently, the distance between the left and right cameras should be sufficient, and the objects should be inside (in the direction of the object) in order to keep the object within the field of view of the left and right cameras.
It is necessary to carry out an operation for directing to, that is, a vergence movement as shown in FIG. 6 (b). Since the above equation (1) does not hold between the images obtained by the two TV cameras TCL and TCR performing the vergence movement, the coordinates on the image are converted by the following equation (2).

However, θ is the angle of convergence of the camera, that is, the angle from the parallel state of the camera to the inside.

FIG. 7 shows an image in which the vergence motion of the television camera is corrected by converting the image of FIG. 5 by the above equation (2). 7 (a) and 7 (b) correspond to FIGS. 2 and 5 (a) and 5 (b).

Next, contour line segment correspondence is performed between the two left and right images. Among the contour line segments obtained from the left and right images, the correspondence that satisfies the epipolar condition of the above equation (1) is obtained. Since this correspondence includes an incorrect correspondence, it is necessary to find a correct correspondence by another method.

The first method is to eliminate the erroneous correspondence by using the characteristics of the region to which the corresponding contour line segment belongs, that is, the average brightness is equal (actually, the difference in brightness is within a predetermined range). Is.

The second method is to group mutually connected contour line segments, perform evaluation between groups, and remove correspondences having low evaluation values.

More specifically, as shown in FIG. 8, the segments S _i and S _i ′, S _j and S _j ′ on the two images correspond to each other, and S
_If the end points of _i and S _j and S _i ′ and S _j ′ coincide, they are defined as “continuous”. Then, search for consecutive correspondences for each correspondence, i.e. segment pair,
A series of consecutive segments is a group.

Next, each group is evaluated. If the entire group is performing incorrect correspondence, the possibility that the endpoints are continuous is smaller and a larger segment (the length of the line segment is longer) corresponds to that when correct response is performed. Possibility is small. Further, when the correct correspondence is performed, the positions of the end points of the corresponding segment pair in the three-dimensional space are not largely displaced in the front-rear direction. Therefore, in order to evaluate those relationships numerically, the evaluation values are defined as follows.

First, the correspondence of each segment that constitutes a group is defined. Segments on different images as shown in Fig. 9
When S _j and S _j ′ correspond to each other, a positive evaluation value C and a negative evaluation value D are defined as follows for this individual correspondence.

C _{j, j ′} = min (▲ ▼, ▲ ▼) D _{j, j ′} = (X _A −X _A ′) − (X _B −X _B ′) where A and B are both end points A of the segment S _j. ′ Is an end point corresponding to A of segment S _j ′ B ′ is an intermediate point corresponding to B of segment S _j ′ D _{j, j ′} is a disparity shift from the start point to the end point of both segments Next, the evaluation value of the entire group G is defined by the following equation (3),
calculate.

When the corresponding segments obviously perform inconsistent correspondence, for example, as shown in FIG. 10, the segments S _j , S _i obtained from the left image and S _j ′, S _i ′ obtained from the right image are obtained. 11 if multiple correspondences that correspond to each other are
As shown in the figure, a combination in which the evaluation value G shows the maximum value is selected using the evaluation formula (3).

By the above processing, the correct correspondence between each segment is selected, and the coordinates of each point on that segment in the three-dimensional space are obtained from the horizontal shift between two images, that is, the parallax, Position measurement is completed.

[Problems to be solved by the invention]

By the way, in the above-mentioned method, after obtaining the corresponding combination between the left and right images by using the epipolar condition, finally, a plurality of segments are grouped based on the continuous relationship between the segments, and the two images are grouped by the group unit. The correct response is sought by conducting an evaluation of. However, this method has the following problems when performing grouping.

First, when performing grouping, the number of combinations of corresponding segments increases, and individual evaluation values are obtained for all of the combinations, so that there is a problem that the processing time becomes long.

Second, when processing a complicated image such as a case where a large number of objects are captured in the input image, the segments that are not actually connected in the three-dimensional space are erroneously connected. However, there is a problem in that the grouping and the evaluation of the correspondence become improper, and an erroneous result is finally obtained.

The present invention has been made in view of the above circumstances, and when performing the grouping of the segments connected to each other on the same image, by adding a new cutting means, erroneous grouping can be performed. It is an object of the present invention to provide an image processing method and an apparatus thereof that can suppress and accurately process a complicated image.

[Means for solving problems]

The image processing method and the apparatus thereof according to the present invention are such that image element elements (contour line segments, pixels, edge points, etc.) obtained from a plurality of images are connected to or close to each other in a pair of elements that satisfy epipolar conditions and the like. Etc. are sequentially extracted, and it is determined whether or not they are on the same plane, and the association between a plurality of images is evaluated only for a group of pairs of elements on the same plane. .

The present invention, in a method of identifying the position on each image obtained by capturing the same recognition target located in a three-dimensional space at different positions, a pixel having a common feature or a line segment connecting pixels from each image. And sequentially extracting pairs of corresponding pixels or line segments between a plurality of images, determining whether or not they form the same plane in the three-dimensional space, and pixels forming the same plane. Alternatively, the position of each recognition target image on each image is specified by evaluating the correspondence between a plurality of images only for a pair of line segments.

[Action]

According to the image processing method and the apparatus thereof of the present invention, a group of image elements (pixels or line segments connecting them) individually extracted from images of the same object captured at different positions and satisfying epipolar conditions and the like. First, it is first determined whether the groups constitute the same plane,
As a result, unnecessary processing is reduced by reducing the number of unnecessary processings and the group including incorrect correspondences is reduced by associating a plurality of images only with respect to groups of image elements that form the same plane. ， Processing efficiency is improved.

〔Example〕

Hereinafter, the present invention will be described in detail with reference to the drawings showing an embodiment thereof. In the following embodiments, the contour segment is extracted as a corresponding unit and evaluated as described in the above [Prior Art]. However, the present invention can directly use pixels, edge points and the like. , Which will be described later.
In addition, the following embodiment is an example in which two left and right images captured by two cameras are processed.

FIG. 1 is a block diagram showing a configuration of an apparatus for carrying out an image processing method according to the present invention, that is, an image processing apparatus of the present invention, which is incorporated in a computer system or the like having an image processing function.

Reference numeral 1 in the figure denotes an image memory in which image data to be processed is stored. The image memory 1 stores two pieces of image data obtained by picking up an image of the same target substance by an image pickup device (not shown) such as a television camera installed at two different positions.
The image data stored in the image memory 1 is given to the image processing means 2 and is replaced after the processing described below is performed.

The image processing unit 2 reads the stored contents of the image memory 1 for each image and performs edge detection, thinning, threshold value processing, full point extension processing, and the like to detect the boundary of the object on the image. Then, the pixels on the obtained boundary points are converted into specific values in the image memory 1 in which the original image is stored. In this embodiment, normal pixel data is represented by a value from 0 to 255, and the pixel at the boundary point is converted into a value of -1.

The area feature extraction means 3 reads the image data from the image memory 1 for each image, detects the boundary point value (−1) written by the image processing means 2, and labels each area divided by the boundary points. Then, each area is identified by its label number, features such as average brightness of each area are extracted, and the results are stored in the area feature storage unit 4.

The area feature storage unit 4 stores the characteristics of each area extracted by the area feature extraction unit 3 described above, and supplies data to the segment correspondence candidate extraction unit 7 as necessary.

In the figure, reference numeral 5 is a contour line segment extracting means, which receives image data from the image memory 1, detects a border line, and for each border line surrounding each region, according to the method described in the section "Prior Art", Divide by the point of maximum curvature of (point sequence) and apply a straight line or curve to each point sequence. Further, when the input television camera or the like is performing a vergence movement, in order to obtain image data equivalent to that when the central axes of the cameras are parallel to each other and at a right angle to a straight line connecting the cameras, Conversion is performed using the equation (1). The fitted line (contour line segment) is sent to the contour line segment storage unit 6 in the form of contour line segment data as shown in FIG. 12 as one unit.

The outline segment data is, as shown in FIG. 12, the sequence number of the segment, the belonging area number, the start point connection segment number (the number of other segments connected at the start point of the segment). , End point connection segment number (number of other segment connected at the end point of the segment), start point neighborhood segment list (segment number near the start point of the segment), end point neighborhood segment list (The number of the segment located near the end point of the segment), the linear type of the segment (whether it is a straight line or a curved line, or the outer or inner line of the area), the segment It is composed of the linear equation of and the coordinates of the start and end points.

The contour line segment storage unit 6 stores the contour line segment data input from the above-mentioned contour line segment extraction means 5, and when there is a request from the segment correspondence candidate extraction means 7 and the group evaluation means 10, it determines the same plane. Means 8
Alternatively, it is sent to the segment group storage unit 9.

The segment correspondence candidate extraction means 7 appropriately extracts the contour line segment data from the area feature storage portion 4 and the contour line segment storage portion 6 and sends it to the same plane determination means 8. The same plane determination means 8 collects the given data into groups forming the same plane and sends the data to the segment group storage unit 9.

The processing contents of the segment correspondence candidate extracting means 7 and the same plane determining means 8 which characterize the present invention, that is, the method of "sequentially extracting corresponding pairs of pixels or line segments" in the claims are as follows.

First, the segment correspondence candidate extraction means 7 stores a first segment register S1 for storing a segment number obtained from an image captured by the first camera and a segment number for the second camera. It has two segment registers S2 and a last-in first-out stack register ST.

First, a value is set to the first segment register S1 sequentially from 1 for each operation. This indicates the segment number obtained from the first camera image (the left image for convenience). Similarly, the second segment register S2 generates the segment numbers of all the segments obtained from the second camera image (referred to as the right image for convenience) each time the content of the first segment register S1 is incremented by 1.

For the segments S _j and S _j ′ designated by the values of the first and second segment registers S1 and S2, the segment correspondence candidate extraction means 7 determines whether the following two "basic correspondence conditions" are satisfied.

The segment S _j is the first camera image and the segment S _j ′ is the second camera image.
It is assumed to have been obtained from each of the camera images of.

Basic correspondence condition: region condition: It is a condition that the features of the regions to which the segments S _j and S _j ′ belong are equal. More specifically, the outline segment data for each segment as shown in FIG. 12 is searched to obtain the number of the area to which it belongs. Next, the contents of the area feature storage unit 4 are searched according to the area number to obtain the characteristics of the area, for example, the average brightness. If the result is within the error of the predetermined range, it is assumed that this condition is satisfied.

Epipolar condition: Provided that there is a combination of points that satisfy the epipolar condition on the segments S _j and S _j ′. More specifically, as shown in FIG. 9, both segments S _j and S _j ′ have a common range in consideration of the error E in the Y-axis direction (upward in the drawing), and that part is In the right image, it is on the left side of the left image. In other words, a certain Y coordinate Y _p
And points (X _p , Y _p ), (X _p ′, Y _p ′) having Y _p as the Y coordinate at the points on the segments S _j and S _j ′ satisfy the following equation.

Y _a −E <Y _p <Y _b + E Y _a ′ −E <Y _p <Y _b ′ + _{EX p} −X _{p b} > −E Segments S _j and S _j ′ satisfy the above basic correspondence conditions, Store S _j and S _j ′ in stack register ST. Next, the following processing is performed toward the starting point direction or the ending point direction of the segments S _j and S _j ′. Here, the starting point direction is the direction of the first point in the point sequence to which the relevant segment is fitted when the boundary line (contour line) is traced by the contour line segment extracting means 5, and the ending point direction is The direction of the last point. The coordinates of the start and end points are
As shown in FIG. 12, it is stored in the contour segment storage unit 6 and is read out by the segment correspondence candidate extraction means 7 at any time.

When tracing in the direction of the start point, first, the segment having the end point that coincides with the start point of the segment S _j within the error E
Search S _i . In other words, the following expression is satisfied when the starting point coordinates of the segment S _j are (Xs, Ys) and the ending point coordinates are (Xe, Ye).

(Xs−Xe) ² + (Ys−Ye) ² <E ² Similarly, a segment S _j ′ having an end point that coincides with the start point of the segment S _j ′ is searched. Further, when the segment S _j ′ and the segment S _j ′ satisfy the basic corresponding condition, it is stored in the stack register ST and all the data in the squak register ST are given to the same plane judging means 8 and as a result, the same plane is obtained. If it is determined not to be above, stack register ST
After deleting the records of the segments S _i and S _i ′ from, a combination of segments to which other conditions apply is searched.

On the other hand, if it is determined that they are on the same plane, for the segments S _i and S _i ′, the other combinations S _k and S _k ′ of the segments S _i and S _i ′ that are respectively connected in the starting point direction and satisfy the basic corresponding condition described above. Search the stack register ST
, And the same processing as that for the aforementioned segments S _j , S _j ′ is performed. Note that here, in consideration of the case where the segments are connected to form a ring, and the like, when the combination of the new segment overlaps with each combination on the stack register ST, it is deleted.

By repeating this process in sequence, the segment
If S _j and S _j ′ correspond to the first segment, then connect them sequentially and satisfy the above basic correspondence condition,
And all combinations of the segments existing on the same plane are sent to the same plane determining means 8.

Further, the first and second segment registers S1 and S2 are sequentially updated. As a result, the segment correspondence candidate extraction means 7
Repeats the above process for all the segments obtained from the first and second images.

When processing three or more N images captured by three or more television cameras or the like, the segment correspondence candidate extraction means 7 similarly uses the first, second ... Nth segment registers S1, S2 ... Sn are used to sequentially generate all combinations, and for each of the combinations (S _i1 , S _i2 ... S _iN ), the combinations corresponding to the basic corresponding condition, connection condition, and same plane condition are sequentially obtained as described above. , And sends it to the same plane determining means 8. In this case, the basic correspondence condition is expanded and the segment
It is assumed that the same conditions as in the case of the two images are satisfied for any combination of S _i1 , S _i2 ... S _iN .

The same-plane determining means 8 is provided with the pair of segment pairs obtained by the segment-corresponding candidate extracting means 7 as described above, reads out the characteristics from the contour segment storage section 6, and makes them the same in the three-dimensional space. It is determined whether or not it is on a plane, and the result is the segment correspondence candidate extraction means 7
When it is determined that they are on the same plane, the segment number of the combination is sent to the segment group storage unit 9.

The details of the processing by the coplanarity determining means 8, that is, the method of "determining whether or not the coplanar plane is formed in the three-dimensional space" in the claims is as follows.

First, the basic principle will be described. Any plane in three-dimensional space
Let ax + by + cz = 1 and let u be the distance between two TV cameras as an image pickup device. Further, when the central axes of both television cameras are parallel to each other and perpendicular to a straight line connecting the cameras, that is, when they are arranged as shown in FIG. 6 (a), the coordinates on the image of the second camera are as follows. It is converted into coordinates on the image of the first camera by the affine transformation matrix as in the formula.

Here, (X, Y) are coordinates in the image when the point P on the plane is seen from the first camera, and (X ′, Y ′) is the point P.
Is a coordinate on the second camera image of.

For example, as shown in FIG. 13, four points on the plane are obtained as points A, B, C, D on the first image, and A ', B', C ', D'on the second image, respectively. It was obtained as. At this time, the following equation holds between each coordinate.

Furthermore, Ya ′ ＝ Ya, Yb ′ ＝ Yb, Yc ′ ＝ Yc, Yd ′ ＝ Yd …… (4a) Will be transformed. That is, a set of points AA ′, BB ′, CC ′, D− obtained by associating two images with each other.
The necessary and sufficient condition that all the points in the three-dimensional space represented by D'are on the same plane is that the Y coordinate of each set is equal as shown in the above equations (4a) and (4b). , And that there are parameters a, b, and c that satisfy the above equation. In the coplanarity determining means 8, the correspondences of the segments sent from the segment correspondence candidate extracting means 7 are pairs that satisfy the epipolar condition, in other words, since the above expression (4a) is satisfied, the expression (4b) is satisfied. It will be sufficient to judge only regarding.

The determination that the pair of segments is on the same plane by the same plane determination means 8 is performed as follows.

A pair of segment pairs (S ₁ −S ₁ ′, S ₂ −S ₂ ′, S ₃ −S ₃ ′, ..., S _n −) input from the segment correspondence candidate extraction means 7
For S _n ′), the coordinates of the starting point and the ending point of each segment are extracted from the contour segment storage unit 6 and stored in a primary storage memory (not shown). The first image segment S _k of the start point of coordinates (X _sk, Y _sk), the end point of the coordinates (X _Ek, Y _Ek) and to the segment on the second image corresponding to the segment S _k S _k ' Similarly, for (X _S ′ _k ,
Y _S ′ _k ) and (X _E ′ _k , Y _E ′ _k ). For all corresponding segments, it is a necessary condition that the pair of start point and end point satisfy the above expression (4b) to form the same plane. Furthermore, when the segment is a straight line, it is a necessary and sufficient condition.

Since the segment obtained from the image has an error such as a quantization error, even if each point is on the same plane in the three-dimensional space, it does not completely match the equation (4b). For that reason,
Predicted intervals are obtained for each point, and if the deviation from the parallax obtained from the image is within a certain threshold TH, it is determined that they are on the same plane. Specifically, the following formula is used so that the value of this formula is minimized, that is, a, b,
Find the value of c.

If the values of a, b, and c are indefinite, it is determined that all points are on the same plane for convenience. Otherwise, confirm that the following equation is satisfied for each segment pair.

_{| (X S1 '-X S1)} -u (-aX S1 + bY S1 + c) | <TH | (X E1' -X E1) -u (-aX E1 + bY E1 + c) | <TH all respects this equation Is satisfied, it is determined that all segment pairs are on the same plane. The coplanar determination means 8 sends the determination result to the segment correspondence candidate extraction means 7. Further, when it is determined that all the segments are on the same plane, the same-plane determination means 8 sends the segment pair which is the determination target to the segment group storage unit 9.

When three or more images are processed, for example, in the case of three images, the first image and the second image, the second image and the third image, the third image and the first image are processed. The same determination is performed between the obtained segments, and only when it is determined that all the respective combinations are on the same plane, the final result is determined to be the same plane.

The segment group storage unit 9 stores a set of segment pairs sent from the same plane determination unit 8 as one group, further stores the evaluation result of the group evaluation unit 10 of the group, and sends it to the selection unit 11. have.

The group evaluation unit 10 sequentially extracts the groups of segments from the segment group storage unit 9, extracts the characteristics of each segment from the contour line segment storage unit, and calculates the formula (3) described in the conventional method for each group. To obtain an evaluation value G, and the result is used as the segment group storage unit 9
Send to.

The selection means 11 searches the storage contents of the segment group storage unit 9 for a group including the same segment of the same image among the groups, and regards only the group having the largest evaluation value G among the plurality of groups as a correct correspondence. Leave and delete other groups. Eventually, all the segments belong to only a single group, and the groups remaining up to that point are sent to the three-dimensional coordinate calculating means 12.

The three-dimensional coordinate calculating means 12 calculates and outputs three-dimensional coordinates for each pair of segments of the group sent from the selecting means 11 based on the principle of triangulation.

The operation of the image processing apparatus of the present invention having the above-mentioned configuration will be briefly described below.

For example, when a stereo image as shown in FIGS. 2 (a) and 2 (b) is input to the image memory 1, edge detection and end point extension are carried out by the image processing means 2 so that the images shown in FIGS. 3 (a) and 3 (b) are obtained. The image is converted into such an image, and the image in which the pixels on the obtained edge points are replaced with a predetermined value (−1) is written in the image memory 1. Next, using the coordinates of the edge points stored in the image memory 1, the area feature is extracted by the area feature extraction means 3.
Is extracted, and sent to and stored in the area feature storage unit 4. Further, the features of the contour line segment are extracted by the contour line segment extraction means 5, the convergence operation is further corrected, and the data as shown in FIG. 7 is stored in the segment feature storage unit 6.

Next, the segment correspondence candidate extraction means 7 operates to make correspondence between contour line segments obtained from each image. Basic correspondence conditions (epipolar condition,
Area conditions, etc.), including consecutive segments,
It is sent to the same-plane determination means 8 as a candidate for a group corresponding to the segment. The coplanarity determining means 8 determines whether or not the affine transformation is established, and when it is established, the given segment group is sent to the storage unit, and the result is sent to the segment correspondence candidate extracting means 7. The segment correspondence candidate extraction means 7 sequentially searches further connected segments and repeats the same processing as long as it is determined that the group to which the data is given to the same plane determination means 8 is on the same plane.

As a result, only the group of segment pairs forming the same plane is recorded in the segment group storage unit 9, and the group of segment pairs as shown in FIG. 14 is formed. In particular, among the candidates output from the segment correspondence candidate extraction means 7, the group including an incorrect correspondence has few continuous segments and is unlikely to be formed on the same plane. Therefore, the groups stored in the segment group storage unit 9 are erroneously associated with a small number of groups, and the erroneously associated groups are composed of a small number of segments. Further, when the correspondence candidates sequentially given from the segment correspondence candidate extraction means 7 to the same plane determination means 8 do not form the same plane, the segment correspondence candidate extraction means 7 stops the search at that stage, so the search target is that. It will be only the amount up to and you can save the dead time.

Next, each group stored in the segment group storage unit 9 is evaluated by the group evaluation unit 10 and selected by the selection unit.
According to 11, the one having the maximum evaluation value G in the formula (3) is selected, and the corresponding group is deleted erroneously. Finally, the three-dimensional coordinate calculating means 12 calculates and outputs the three-dimensional coordinates of each segment pair forming the group.
FIG. 15 shows a three-dimensional image obtained from FIG. 7, that is, an image (a), a front image (b), a side image (c), and a plane image (d) viewed from the center points of both cameras.

In the above embodiment, the segment in which the pixels are connected is used as the corresponding unit, but when the pixel or the edge point is directly processed as the corresponding unit, the processing is performed by the device having the configuration shown in the block diagram of FIG.

Compared to the device shown in FIG. 1 which uses the contour line segment as a corresponding unit, this device configuration deletes the reference numerals 3, 4, 5 and 6 for extracting and storing the region and the contour line segment, and others. This is a device with some changes. Reference numerals of the respective parts are given corresponding to FIG. Here, a brief description will be given focusing on the points different from the configuration of FIG.

The image input to the image memory 1 is processed by the image processing means 2. In the case of an edge point, processing such as edge extraction, thinning, and end point extension is performed as in the case of using the contour line segment. Further, in any case, the convergence operation of the camera is corrected.

Next, the point correspondence candidate extraction means 17 (corresponding to the segment correspondence candidate extraction means 7 in FIG. 1) operates to search for a pair of points satisfying the epipolar condition among a plurality of images. Further, a pair of points located near and corresponding to the point is searched for and sequentially sent to the coplanar determination means 8. The point that the search is stopped or continued according to the determination result is the same as in the above-described embodiment.

The coplanarity determining means 8 is modified so that the part using the start point and the end point in the apparatus for processing the contour line segment of FIG. 1 uses the coordinates of the given point itself, and only when each point composes the same plane. It is recorded in the point group storage unit 19 (corresponding to the segment group storage unit 9 in FIG. 1).

The group evaluation means 10 compares, for each group, the characteristics of a pair of points forming each group, and the characteristics of brightness, edge strength, etc. are close for each pair of points,
In addition, a higher evaluation value G is given to a pair having a larger number of pixels and points in the group. The result is recorded in the point group storage unit 9 for each group.

The selection means 11 compares the evaluation values G between the groups including the same pixel or edge point, and deletes the groups other than the group having the maximum value. The group remaining as a result is sent to the three-dimensional coordinate calculating means 12, and the coordinates of the pair of points forming the group in the three-dimensional space are calculated and output.

In the above-described embodiment, the method of determining the three-dimensional coordinates by performing the correspondence between the images using the contour line segment, the pixel, and the edge point as the correspondence unit has been described. However, the present invention provides points (2) obtained by other methods. Needless to say, it is also applicable to the case where the unit is a segment obtained by tracing the point, such as the second derivative zero crossing).

〔effect〕

As described above, according to the present invention, in the case where the association is performed in the unit of the conventional segment or pixel and the grouping is performed, the processing time is required because the number of combinations is large, and the reliability of the association is low. As long as the pairs forming a group form the same three-dimensional plane at the time of search, the search is limited and the number of target combinations is reduced. Therefore, according to the present invention, the processing time is shortened, and since there is a high possibility that the wrong means will delete the group because the group becomes smaller, a more reliable correspondence can be obtained.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an apparatus for carrying out the method of the present invention with a contour segment as a corresponding unit, and FIGS. 2 (a) and 2 (b) are images obtained by a stereo camera. ) Shows the left screen and (b) shows the right screen.
2A is an example in which edge detection and thinning are performed on the right screen in FIG. 2B, FIG. 4B is an example in which end points are further extended, and FIG. 4 is a flowchart showing a method for creating a contour segment. FIG. 5 is an example in which contour line segments are created, FIG. 6 (a) is a schematic view showing the positional relationship of cameras installed in parallel, and FIG. 5 (b) is a schematic view showing the positional relationship of cameras performing vergence movement. 7 (a) and 7 (b) correct the vergence movement, FIG. 7 (a) shows the left screen, and FIG. 7 (b) shows the right screen. FIG. FIG. 9 is an explanatory diagram of the correspondence relationship of segments, FIG. 10 is an explanatory diagram of multiple correspondence, and FIG.
The figure is an explanatory diagram of the method to select the correct correspondence from multiple correspondence,
Fig. 12 is a schematic diagram of the data representation of contour line segments, Fig. 13
The figure is an explanatory diagram of the groups of the corresponding segments between the two screens, and Fig. 14 is the correspondence example of the groups actually obtained, Fig. 15
FIG. 16 is an example of a finally obtained three-dimensional image, and FIG. 16 is a block diagram showing a configuration of an apparatus for carrying out the method of the present invention by using pixels or edge points as corresponding units. DESCRIPTION OF SYMBOLS 1 ... Image memory, 2 ... Image processing device 7 (17) ... Segment correspondence candidate extraction means (point correspondence candidate extraction means), 8 ... Same plane determination means 10 ... Group evaluation means

Claims (1)

[Claims] 1. An image processing method for identifying a position on each image obtained by picking up the same recognition target located in a three-dimensional space at different positions, wherein pixels or pixels having common characteristics are identified from each of the images. Obtains connected line segments, sequentially extracts pairs of corresponding pixels or line segments between multiple images, determines whether or not they form the same plane in three-dimensional space, and forms the same plane. An image processing method characterized in that the position of each recognition target image on each image is specified by evaluating the degree of correspondence between a plurality of images only for a pair of pixels or line segments that are being processed.