JP3343583B2 - 3D surface shape estimation method using stereo camera with focal light source - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for estimating a three-dimensional shape using a stereoscopic visual sensor, and more particularly to a method for estimating the shape of a planar or curved object using a stereo camera with a focal light source.

[0002]

2. Description of the Related Art When a robot moves in a room, it is necessary to recognize objects placed in the room and to move avoiding these objects. For this purpose, a visual sensor for grasping the object is installed in the robot. ing. Conventionally, as a method of estimating a three-dimensional shape by measuring the distance to an object using a visual sensor, a stereo stereoscopic method using the principle of triangular distance measurement, a method using ultrasonic waves and a laser range finder are conventionally used. It had been.

Referring to FIG. 10, the principle of a method for estimating the three-dimensional shape of an object by stereoscopic vision will be described. In this method, two visual sensors such as cameras are arranged at right and left positions R and L to view a stereo image. Since the positions R and L of the camera are determined, the vertex of the triangle and the point P of the target object are uniquely determined from the side angles a and b of the side a. Camera posture,
Since the focal lengths of the lenses are known, both end angles b and c can be obtained from points P ₁ and P _{2 on} the right image plane and the left image plane.

[0004]

The conventional method of applying the stereoscopic stereoscopic method by triangulation is simple and simple because only two cameras are used. However, in order to determine the line-of-sight direction of the camera, it is necessary to find out where the visible point in the right image corresponds to the left image. It is difficult to automatically find corresponding points from two input images. The computer can determine the corresponding point only in the part of the object where the brightness of the surface changes suddenly, so for an object with a uniform surface without features, perform three-dimensional measurement of the entire surface There was a problem that it was not possible.

Further, the conventional three-dimensional shape estimation method using an ultrasonic wave or a laser range finder requires a special device such as a laser range finder or an ultrasonic device. In the case of using a laser range finder, it took time to calculate the shape of the target object, and in the case of using an ultrasonic device, the calculation time was short but the accuracy was poor.
On the other hand, when the robot moves indoors avoiding obstacles, similar to the human eye, the three-dimensional information of the objects necessary for the robot is sufficient only with the approximate position and size of the obstacle object. The calculation time for grasping the obstacle must be shorter than the movement time, and these visual sensors cannot be said to be sufficient as the visual sensors of the mobile robot.

[0006]

In order to achieve the above object, according to the present invention, an annular light source is equivalent to a surface of a target object.
A plurality of cameras with a focal light source, comprising a plurality of cameras with a focal light source, and a circular light source illumination provided coaxially with the optical axis of the camera at a distance that can be regarded as a point light source, and a stereo camera with a focal light source, comprising a CPU. Using the two light sources, the target object image based on the reflected light of the target object by one illumination of both light sources is captured by both cameras, the CPU captures these video signals, finds the regular reflection point in both images,
Provided is a method of calculating three-dimensional coordinates of one point of a target object plane located in front of a camera and normal directions of three points, and approximating the target object plane to estimate a three-dimensional surface shape.

According to the method of the present invention, it is possible to estimate a target object surface having a smooth, almost vertical curved surface without feature points in a short time by using a commercially available simple device such as a ring light source, a camera, and a CPU. it can.

[0008]

FIG. 1 shows an example of an apparatus for carrying out the method of the present invention. 1R and 1L are right and left cameras with focal light sources constituting a stereo camera, 2R and 2L are right and left light receiving devices, for example, cameras, 3R and 3L are targets.
An annular light source can be equivalently regarded as a point light source for the surface of a property
Right and left annular light source devices provided coaxially with the optical axis of the camera at a distance , 4 is an analog / digital converter, 5 is a CP.
U and 6 represent monitors, 7R and 7L represent irradiation light, and 8R and 8L represent reflection light of the irradiation light 7L and 7R from the object surface.

The annular light source devices 3R and 3L are, for example, annular strobes, and are mounted coaxially with the position lenses at the projection centers of the cameras 2R and 2L. In general, an annular light source becomes equivalent to a point light source when the distance to a target surface is 10 times or more the diameter of the light source. The stereo camera with the focal light source shown in FIG.
The annular light source devices 3R and 3L are coaxial with the optical axis of the camera, and the center of the annular light source is equivalent to a point with respect to the surface of the target object.
Because it is arranged at a distance which can be regarded as a light source, focusing of the camera
Functions as a point light source located at a point position (hereinafter, circular
The light source device is simply called a point light source device. ) . FIG. 2 is a perspective view of a mobile robot in which a stereo camera with a light source including the point light source devices 3R and 3L and the cameras 2R and 2L is arranged at the upper part.

The analog / digital converter 4 is a camera 2
The image captured by R and 2L is sampled, each pixel is digitized at a luminance level, and transmitted to the CPU 5. CPU5
Controls the cameras 2R and 2L and the light source illuminations 3R and 3L, stores the video signal from the analog / digital converter 4, and calculates three-dimensional information.

The method of the present invention will be described with reference to the flowchart of FIG. 3 with reference to the stereo camera with the focal light source of FIG. Step S1 First, one of the left and right point light source devices 3L and 3R of the stereo cameras with focus light sources 2R and 2L, for example, the right point light source device 3R is first turned on, and the target object is photographed by the cameras 2L and 2R. Next, the left point light source device 3L is turned on and the cameras 2L and 2L are turned on.
The target object is photographed with R. The video signals of these four images are converted from analog to digital and input to the CPU 5. FIG. 4A is an example of a right image captured by the right camera 2R when the right point light source illumination R is turned on.

Step S2 The CPU 5 extracts an object region from each input image,
Extract the isoluminance curve. First, the Sobel operator (Sobe
l operator) and Region labeling
Use ling) to extract the surface area. The Sobel operator examines the change in the luminance around the pixel, and the pixel with a small change is defined as the surface portion, and the other pixels are defined as the edge portion.
Into two areas. Regarding the plane region, the connectivity of pixels is checked by region labeling, and a region having an area equal to or larger than a certain value is determined as an object region.

Next, in order to remove uneven brightness due to the material of the target object surface or the minute surface shape, the brightness of each surface area is averaged by a 7 × 7 window operator. Further, the luminance averaged for each plane is quantized for each fixed level, and a connected region having the same luminance is obtained by region labeling. This connecting area becomes a strip,
More precisely, this area is thinned to obtain an isoluminance curve.

FIG. 4B shows an example of a right image captured by the right camera 2R when the right point light source device 3R is turned on (FIG. 4A).
3 is a diagram showing an example in which an extracted object region image and an isoluminance line image are placed thereon. The object region from which the region surrounded by U1 'to U4' is extracted shows an isoluminance curve in which E is calculated. In this example, the brightness is set to 2 for convenience of display.
It is an equal luminance curve thinned out for each level. Since the peak of the luminance distribution is a specular reflection point, the CPU 5
Calculates the peak position of the luminance distribution based on the calculated concentric circle group of the equal luminance curve and the center point of the elliptic curve group, and calculates the position of the regular reflection point. In the isoluminance curve in the luminance distribution based on the regular reflection light component, the minor axis of the elliptic curve is about 10 to 60 pixels.
If the average value of the length of the short axis is 60 pixels or less, this luminance distribution may be a regular reflection light component.

Step S3 In an image obtained by photographing the object illuminated by the point light source, the three normal directions and one-dimensional information at one point at the peak position with respect to each local region where the reflected light peaks are described below. It is calculated by:

FIG. 5A shows a light irradiation path and a reflection path when the left point light source 3L is turned on, and a local area (image) where the reflected light on the image plane captured by the left and right cameras 3L and 3R has a peak. FIG. 5 shows an area represented by an ellipse in the plane).
(B) shows a light irradiation path and a reflection path when the right point light source 3R is turned on, and a local region where the reflected light on the image plane captured by the left and right cameras has a peak. In FIG. 5, points A and C of the object are specular reflection points observed on the same side, for example, in the left camera image when the left point light source 3L is turned on, and the image has the highest luminance. Point B is a point where the luminance peak value is observed in both the right camera image and the left camera image when the left point light source is turned on and when the right point light source is turned on.

The origin of the three-dimensional coordinates is set at the midpoint of the baseline of the stereo camera (the point indicated by O in FIG. 5), the direction of the line of sight of the camera is the Z axis, and the direction perpendicular to the plane perpendicular thereto is Y.
A right-handed coordinate system with the axis and horizontal direction as the X axis. L and R in FIG. 5 represent the position of the left point light source 3L and the focal position of the left camera 2L, the position of the right point light source 3R, and the focal position of the right camera 2R, respectively. Let the focal coordinates of the left and right cameras be the following coordinates (1).

In FIG. 5, the observation three-dimensional position corresponding to the point B on the left and right camera image planes, that is, the luminance peak observation on the left and right camera image planes when the right and left point light sources are turned on. Assume that the positions are measured as in the following (2). Then, the three-dimensional coordinates (B _x , B _y , B
_z ) is the following equation (3), (4), or (5). The vector in the normal direction nB at point ^B can be expressed by the following equation (6).

[0019]

(Equation 1)

[0020]

(Equation 2)

The point A in FIG. 5, the vector of the normal direction n ^A of the point C, the vector of n ^C is observed three-dimensional position, i.e. left, left when the lights on the right of point light sources, with the right image plane Assuming that the peak observation positions of the following are respectively (7), the values of the following equations (8) and (9) are obtained. It is to be noted that the same reference numerals in the specification denote the same or corresponding components.

[0022]

(Equation 3)

Step S4: The three normal directions of points A, B and C of the target object image and B
Using the coordinates of the point, the epipolar surface (Epipola
r plane) is approximated by various approximation curves. An example of approximate estimation using two arcs S and S 'will be described with reference to FIG.

The reason for approximation by a circular arc is based on the following two points. Be able to express a smooth curved surface with few parameters. Artificial surfaces that actually exist indoors are often created based on arc surfaces such as spheres and cylinder sides, and have good approximation accuracy.

First, an intersection O _AB between two lines (see FIG. 6) passing through the points A and B is determined, and this is defined as the center of the sphere. Actually, there is an error. If the two lines passing through the point A and the point B cross three-dimensionally and the two lines do not intersect, the points P1 and P2 on both lines where the two lines approach the shortest are obtained. , Let these midpoints be the centers of the spheres. P1 on two lines passing through these points A and B
The coordinates of the point P2 and the coordinates of the point P2 are expressed by the following equation (10) and (1
1) It is obtained by the equation. Therefore, the center SO ^AB and the radius SR ^AB of the sphere for approximating the cross section of the object surface with an arc are given by the following (1).
2) and (13).

[0026]

(Equation 4)

[0027]

(Equation 5)

The cross-sectional shape of the object surface is obtained by finding the intersection with the sphere for each pixel existing on a straight line connecting the luminance peak points on the image corresponding to points A and B on the object surface. The three-dimensional position of each pixel on the image plane is expressed by the following (14).
Then, the intersection of this with the sphere is (15).
The intersection between the point B and the point C on the object surface and the sphere can be calculated in the same manner.

[0029]

(Equation 6)

Step S5 A reflection map of the incident angle on the target surface and the luminance of the surface is estimated from the luminance of the pixel corresponding to the surface of the cross section on the grayscale image. The luminance (shading value) of each pixel is determined by the position of the light source and the camera, the reflection coefficient of the object surface, and its inclination.
Use this to shape the remaining normal directions in that area.
The method is determined by -From-Shading (a method of extracting the direction of the object surface from the grayscale image and reconstructing a three-dimensional shape).

1. A method of calculating the angle of incidence at each point on the surface of the object cross section estimated in step S4 and obtaining the reflection function of the target object surface will be described with reference to FIG. The coordinates of the Pi point on the object surface in the cross section are (Xi, Yi, Zi) (i = 1 to
n), the normal direction n ^p of the target object plane at this Pi point
Is the center S0 of an arc approximating the surface of the cross section.
Since the direction connects (S0x, S0y, S0z) and this Pi point, it can be expressed by the following equation (16). Vector of the light source direction Ln ^p in the Pi point when using the left point source 3L is determined by the following equation (17). Therefore, the incident angle θp can be obtained by Expression (18).

[0032]

(Equation 7)

Plotting the relationship between the thus obtained incident angle θp at each point on the cross section of the object surface and the luminance of that point on the observation image (when the left point light source device 3L is lit in FIG. 7A). 7 (b) are obtained. This is the reflection function representing the reflection characteristics of the observation surface.

2. Next, a method of calculating three-dimensional coordinates corresponding to each pixel point of the observation image other than the estimated shape section will be described with reference to FIG. First, from the luminance of each pixel, 1. The incident angle θ at a point on the target observed by the pixel is obtained using the reflection function obtained in the above. Thereby, for example, FIG.
Pixel _{(i n, j n + 1} ) such observation image as shown in the vector of the camera line of sight on the conical surface of the top angle θ of the center line in the normal direction n point P on the object plane for the pixel Limited. From the condition that the surface of the target object is smooth,
The nearest normal vector of the normal direction n ⁿ points in the vicinity of known calculated from the conical side surface, and a normal line at the point on the corresponding target surface.

This can be specifically determined as follows. The three-dimensional coordinate system is translated so that the vertex O ′ (see FIG. 8B) of this cone (the side surface length is defined as unit 1) is set as the origin. Now, the vector of the normal direction n at point P on the object surface to be obtained, when the intersection of the vector and the bottom surface of the cone of n ⁿ was placed at the origin O 'and ^a point P, which circle the bottom it is a vector of the intersection P ² points endpoint of the direction of the circumference extending in the radial direction. The conical bottom surface can be expressed by the following equation (19) using a known V0 vector (negative vector in the line-of-sight direction) in the direction of the point light source. The intersection of this and the straight line expressed by the following equation (20) is It is obtained by the following equation (21). Therefore, the vector in the normal direction n to be obtained is obtained by the following equation (22).

[0036]

(Equation 8)

[0037]

(Equation 9)

Here, the three-dimensional coordinate system is returned to the original origin O.
Translate as follows. Pixel (i_n, J_{n + 1}Corresponding to
The point P on the surface of the object
And a known neighboring point whose three-dimensional coordinate value is (23)
PⁿFrom the three-dimensional coordinate value (P _x, P_y, P_z)
The vector to P and the normal vector (n_x,
n_y, N_z) Are orthogonal to each other, so that
Is calculated.

[0039]

(Equation 10)

As described above, the three-dimensional coordinates P of the surface of the target object are calculated by sequentially obtaining the normal vectors of the neighboring points from the normal vector of the sectional position of the shape, and the 3D coordinates of the target object surface are calculated.
The shape can be estimated.

FIG. 9 shows an example of estimating a surface shape in which the target object is a planar object U and a cylindrical object V by the method of the present invention. R and L in FIG. 9 represent the right and left point light source positions and the focal position of the camera. In this example, the object area was calculated from the left image. Therefore, the object area is directed to L. There is little difference even if the calculation is performed using the right image. FIG.
(A) is a calculation example of step 3 according to the present invention, (b) is step 4 and the left camera 2 at the position L obtained in step 2 is shown.
Left and right visual ranges U1 'to U2', V of the object area viewed from L
This is an example of calculating a surface cross section approximated by an arc in the range of 1 ′ to V2 ′. (C) is an example in which a reflection function is obtained from a graph in which the incident angle and the luminance observed on the image plane are plotted at points on the surface of the cross section estimated in step 4. (D) shows how Shap is obtained from the observed image in the object areas of the planar object U and the cylindrical object V, which are the target objects obtained in step 2.
This is an example in which the 3D shape surfaces of a planar object and a cylindrical object are estimated by performing calculations using the e-From-Shading method. In the above, the method of the present invention has been described for two cameras with a focal light source. However, the method of the present invention is possible even if there are three or more cameras with a focal light source.

[0042]

In contrast to the conventional stereoscopic binocular stereoscopic method using two cameras which cannot sufficiently perform three-dimensional reconstruction on a small area having no features such as edges, the present invention is not limited to this. Is a method of measuring the luminance distribution by actively illuminating with a simple device such as a camera and a focal light source added to the camera, and obtaining three-dimensional position information of one point on a surface having no feature and a method of three points. Calculate line directions easily, and then use Shape-F
The three-dimensional shape of the surface can be calculated using the principle of the rom-Shading method.

The peak position of the luminance distribution is less affected by disturbance, and its position can be estimated by simple image processing.
INDUSTRIAL APPLICABILITY The method of the present invention can obtain position and shape information of a surface at a distance of up to several meters, and is suitable for a vision system of a robot moving in a building.

[Brief description of the drawings]

FIG. 1 is a conceptual diagram of a stereo camera with a focal light source for implementing the method of the present invention.

FIG. 2 is a perspective view of a mobile robot provided with a stereo camera with a focal light source.

FIG. 3 is a flowchart illustrating the method of the present invention.

FIG. 4 is a diagram illustrating functional steps 1 and 2 of the present invention.

FIG. 5 is a diagram illustrating functional step 3 of the present invention.

FIG. 6 is a diagram illustrating functional step 4 of the present invention.

FIG. 7 is a diagram illustrating a method for obtaining a reflection function of a target object surface in function step 5 of the present invention.

FIG. 8 is a diagram illustrating functional step 5 of the present invention.

FIG. 9 is a diagram illustrating an operation example of the present invention.

FIG. 10 is a diagram illustrating the principle of stereo stereoscopic viewing.

[Explanation of symbols]

1R, 1L Camera with focal light source 2R, 2L Light receiving device, camera 3R, 3L Ring light source device 4 Analog / digital converter 5 CPU 6 Monitor 7R, 7L Irradiation light 8R, 8L Reflection light

──────────────────────────────────────────────────続 き Continuation of front page (58) Field surveyed (Int.Cl. ⁷ , DB name) G01B 11/24 G06T 7/00

Claims (1)

(57) [Claims] 1. An annular light source for a camera and a surface of a target object
Is equivalent to a point light source and is the same as the optical axis of the camera.
Using a plurality of cameras with a focal light source comprising an annular light source illumination provided on a shaft, and a stereo camera with a focal light source comprising a CPU, both light sources are sequentially turned on and based on respective reflected lights of the target object. The target object image is photographed by both cameras, the CPU captures these video signals, finds the specular reflection points in both images,
3. A stereo camera with a focal light source, which calculates three-dimensional coordinates of one point of a target object plane located in front of the camera and normal directions of the three points and approximates the target object plane.
Dimensional surface shape estimation method.