JPH0792370B2 - Object and light source information extraction method and apparatus using the same - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object and light source information extraction method and an apparatus using the same. More specifically, the present invention constructs a state of reflectance and shape of a three-dimensional object or brightness and direction of a light source in a computer by image processing, and automatically inputs an object model and a mobile robot. And a light source information extraction method and an apparatus using the same, which are used for environment recognition, etc., in the field of measurement including FA, etc., and environment recognition in mobile robots, etc.

[0002]

2. Description of the Related Art A method for extracting information on a three-dimensional object from a distribution of image intensity is called "shape restoration from shadow". FIG. 8 shows a sphere 8 as an example of an object in order to show its outline.
1 is used. When a three-dimensional object whose surface has a uniform diffuse reflection is illuminated by a point light source 82 that is sufficiently distant, the image intensity at each coordinate point of the captured image is uniquely determined by the orientation of the surface of that point. Therefore, if the brightness and direction of the light source are detected by some method, it is possible to limit to some extent which direction the surface of the point is facing from the strength of the imaged image.

A specific description will be given below with reference to an example. Figure 9
Is a rewriting of the brightness distribution of FIG. 8 with a curve connecting points of the same brightness. For example, when a light source 82 is arranged as shown in FIG. 9 and the image intensity of a certain imaged point (i, j) having the same reflectance as the sphere 81 is 0.2, the direction of the plane of that point Is limited to the set of face orientations on the 0.2 brightness curve indicated by the thick line 91 on the sphere. But,
With this method, it is not possible to narrow down further, and the orientation of the surface cannot be uniquely determined.

As a method for solving this problem, the positions of the camera and the object are fixed, and the light source is successively irradiated from three different directions to extract the information of the three-dimensional object from the distribution of the intensity of the three images. A photometric stereo method has been proposed.

FIG. 10 is a diagram for explaining the principle of the above-described photometric stereo method. In FIG. 10, it is assumed that the brightness of the light source in three directions is uniform or known, and the directions thereof are obtained in advance. First, the image intensity of a point (i, j) only by the illumination of the light source 101 is I ₁ (i, j) =
If it is 0.2, the direction of the surface is indicated by the thick line 10 as in FIG.
Can be limited to above 4. Furthermore, the image intensity at that point due to only the illumination of the light sources 102 and 103 is I
_{If 2} (i, j) = 0.8 and I ₃ (i, j) = 0.2, it can be limited to the thick lines 105 and 106, respectively. Therefore, it can be seen that the plane orientation of the imaged point (i, j) is uniquely represented by the plane orientation at the intersection 107 of these thick lines. Further, if the direction of the surface is obtained, the direction of the light source has already been given, so the relative reflectance at that point can also be determined from the value of the image intensity.

[0006]

As described above, the photometric stereo method is an excellent method for uniquely determining the orientation and reflectance of the surface of a three-dimensional object, but has the following drawbacks.

First, the brightness of the three light sources is uniform or known, and it is necessary to obtain their directions with high accuracy. Next, it is impossible to obtain the surface orientation and reflectance of the coordinate point that would make even one of the three captured images a shadow.

Therefore, the main object of the present invention is to provide the relative brightness of a three-dimensional object without giving the brightness and the direction of the light source in advance, and even if there are shadowed parts in a plurality of picked-up images. An object information extracting method for uniquely determining the direction and reflectance of a flat surface and an apparatus using the same.

Another object of the present invention is to provide a light source information extraction method for extracting the relative brightness and direction of a light source, and a device using the same, since the brightness and direction of the light source are not given in advance. That is.

[0010]

The invention according to claim 1 is
By processing a plurality of images of an object whose shape illuminated by different unknown light source directions is unknown, the relative reflectance and surface orientation of the object and the relative brightness and light source direction of the light source are determined. A method of extracting object and light source information, comprising: a first step of forming an image matrix composed of image intensity data from each image; and two sets of surface characteristic matrices using the singular value decomposition of the formed image matrix. And the light source characteristic matrix are decomposed into a product form, and the second step of selecting one set and the processing result of the second step indicate that the brightness of the light source is constant or the reflectance is constant or its ratio is constant. A predetermined transformation matrix is calculated according to the known condition, and the transformation matrix is applied to the processing result of the second step to obtain the relative reflectance and the surface orientation of the object and the relative light source brightness. Sheath light source direction Configured to include a third step of extracting.

According to a second aspect of the present invention, as a preprocessing of the second step of the first aspect of the invention, a step of setting an area on the image plane coordinates, and all image intensities of the image in which the set area is a shadow It includes the steps of deleting the data from the image matrix and the step of constructing an image submatrix that does not include the shadow area only with the image intensity data of the coordinate points that do not always produce shadows from the image matrix. Comprising a step of calculating a coordinate point whose relative reflectance and surface orientation are unknown from the image sub-matrix including the area and the obtained relative brightness and direction of the light source.

According to a third aspect of the present invention, a plurality of images of an object whose shape is unknown illuminated by different unknown light source directions are processed, and the relative reflectance and surface orientation of the object are processed. And an object and light source information extraction device for extracting relative brightness and light source direction of a light source, and image pickup means for picking up an image of the object, and light source moving means for changing the relative positional relationship between the object and the light source. , An image matrix consisting of image intensity data from images from different light source directions picked up by the image pickup means is decomposed into a form of a product of two sets of surface characteristic matrix and light source characteristic matrix using singular value decomposition, and 1 After selecting a set, calculate the predetermined transformation matrix according to the condition that the brightness of the light source is constant or the reflectance is constant or the ratio is known from the processing result, and apply the transformation matrix to the processing result. And by configured to include an image processing means for extracting the brightness and the light source direction of its orientation relative reflectance and the surface of the object and the relative light sources.

[0013]

The object, the light source information extracting method, and the light source information extracting apparatus according to the present invention are composed of image intensity data from each image of an object whose shape is unknown illuminated by different unknown light source directions. Make up the matrix,
The image matrix is decomposed into the product of two sets of the surface characteristic matrix and the light source characteristic matrix by using singular value decomposition, and one set is selected, and from the processing result, the brightness of the light source is constant. Alternatively, a predetermined conversion matrix is calculated under the condition that the reflectance is constant or the ratio is known, and by applying the conversion matrix to the processing result, it is possible to calculate the three-dimensional shape without giving the brightness or direction of the light source in advance. It is possible to accurately determine the relative surface orientation and reflectance of an object.

Further, in terms of not giving the brightness and the direction of the light source in advance, from the plurality of images of the object illuminated by the different light source directions, the relative brightness and direction of the light source are obtained. Can be extracted.

Furthermore, in any of a plurality of images obtained by picking up an object illuminated by different light source directions, if any one of them has a shadow region, the decomposition by the singular value decomposition and the selection of the decomposition result are performed. In the process, as a pre-process, a region is set in the image plane coordinates, all image intensity data of the image in which the set region is a shadow is deleted from the image matrix, and the shadow is always cast from the resulting image matrix. The image small matrix that does not include the shadow area is constructed only by the image intensity data of the coordinate points that do not become, and as a post-processing, the image small matrix that contains the remaining shadow area and the obtained relative light source brightness. By calculating coordinate points whose relative reflectance and surface orientation are unknown from the sheath direction, the relative surface of the three-dimensional object can be obtained even if there is a shadow in a plurality of captured images. Orientation It can be determined with high reflectivity accuracy.

[0016]

1 is a schematic block diagram of an embodiment of the present invention. In FIG. 1, a cylinder 1 as an example of an object is illuminated by a light source 2 and a television camera 4 as an image capturing unit captures an image of the cylinder 1. The image captured by the television camera 4 is given to the image processing device 5 and processed. The position of the light source 2 relative to the object is controlled by the moving device 3. However, since the movement of the light source 2 does not require accuracy, it may be manually moved appropriately or a light source such as the sun that cannot be controlled may be used.
The image processing device 5 and the moving device 3 are controlled by the CPU 6, and the CPU 6 executes a process for extracting the object information of the cylinder 1 and the light source information of the light source 2.

2 to 7 are flow charts for explaining the operation of one embodiment of the present invention. In particular, FIG. 2 shows the overall flow, and FIG. 3 shows the steps shown in FIG. SP
5 shows a specific flow of SP1 and FIG. 4 shows a specific flow of step SP2.
Shows a specific flow of step SP3. Furthermore, FIG. 7 shows a specific flow of step SP2 in the case where a shadow region exists in any one of a plurality of images of the cylinder 1 illuminated by different light source directions.

The specific operation of the embodiment of the present invention will be described below with reference to FIGS. The television camera 4 shown in FIG. 1 captures an image of the cylinder 1 under the illumination of the light source 2 and supplies the image output to the image processing device 5. The image processing device 5 processes the captured image and gives it to the CPU 6.
The CPU 6 outputs a command signal to the moving device 3 to move the light source 2 arbitrarily. By repeating these operations, the television camera 4 sequentially takes images of the cylinder 1 under different light source directions.
If the light source 2 is manually moved or a light source such as the sun that cannot be controlled is used, a plurality of images may be taken continuously or at appropriate intervals while moving or moving the light source. Good.

In step SP1 shown in FIG. 2, the CPU 6 takes in each image signal from the image processing device 5 and generates an image matrix. Then, in step SP2, the CPU 6 decomposes the image matrix into a product of two sets of two matrices by using singular value decomposition, and selects one set. Further, the CPU 6 proceeds to step SP3 at step SP3.
From the processing result of No. 2, an appropriate conversion matrix is calculated according to the condition regarding the brightness of the light source 2 or the reflectance of the cylinder 1, and the conversion matrix is applied to the processing result of step SP2. If any one of the plurality of images of the cylinder 1 illuminated by different light source directions has a shadow area, pre-processing and post-processing are added in step SP2.

The operation of each of the above steps will be described in more detail. The specific processing of step SP1 shown in FIG. 2 is shown in FIG. Step SP11 shown in FIG.
At, the index variable k is initialized to 0. In step SP12, the image signal from the television camera 4 is read into the image processing device 5, and the image data image-
k. It should be noted that one imaged image has an image size having n (where n ≧ 6) coordinate points, and the intensity of the image captured when the light source 2 is not lit when forming the image data. The value of is subtracted from the captured image intensity. The image data image-k thus obtained is given to the CPU 6.

Step SP13 is a process for picking up an image of the cylinder 1 under illumination of different light source directions.
6 issues a command to the moving device 3 to move the light source 2. Accordingly, the index variable k is incremented in step SP14. CPU6 is step SP
15, the index variable k is m (where m ≧
6) and whether the index variable k is m
If not, the process returns to step SP12 and the above step S
The process of P12 to 14 is repeated.

The process is repeated until the index variable k reaches m, and m image data having n coordinate points are obtained. Then, at step SP16, the image matrix I shown in the following equation (1) is obtained. To generate. Note that each row of this matrix corresponds to each coordinate point of the image, and each column corresponds to each image under illumination in one light source direction.

[0023]

[Equation 1]

Finally, in step SP17, it is confirmed by calculation that the rank of the obtained image matrix I is 3 or more, and the operation of step SP1 is ended. When the number of floors is 2 or less, the processing from step SP2 is not performed, an error message is output, and the entire operation ends.

The image matrix I generated in the above step SP1 satisfies the following relational expression (2) by approximating the surface of the observed object as a diffuse reflection surface.

[0026]

[Equation 2]

The above N is called a surface characteristic matrix, and S is called a light source characteristic matrix. A row vector of the surface characteristic matrix N is called a surface characteristic vector for each coordinate point, its magnitude corresponds to reflectance, and its direction corresponds to the normal direction of the surface. Further, the column vector of the light source characteristic matrix S is called a light source characteristic vector for each image, and its size corresponds to the brightness of the light source and its direction corresponds to the light source direction. Step S described below
The image matrix decomposition 1 of P2 and the image matrix decomposition 2 of step SP3 are processes for uniquely obtaining such a surface characteristic matrix N and a light source characteristic matrix S.

Next, referring to FIG. 4, the processing of the image matrix decomposition 1 in step SP2 will be described. First, in step SP21, when the image matrix I is n ≧ m, singular value decomposition is applied to the image matrix I to obtain
The following non-negative diagonalization matrix Σ, (m, m) matrix V of (n, m) matrix U, (m, m) is obtained. When n <m, the transposed matrix may be calculated in advance, and the surface characteristic matrix N and the light source characteristic matrix S may all be replaced with each other to perform the same processing.

[0029]

[Equation 3]

Further, from the three matrices U, Σ and V obtained by the singular value decomposition, the respective submatrices U ′, Σ ′,
V'is extracted as follows.

[0031]

[Equation 4]

Then, in step SP22, two sets of temporary surface characteristic matrix N'and temporary light source characteristic matrix S'are calculated as follows.

[0033]

[Equation 5]

Then, in step SP23, one set is selected from the products of these two sets of matrices. As a precondition for making this selection, the relative relationship between three light source direction vectors that are not on the same plane or the normal vector of the surface must be known. The relative relationship here is an arbitrary coordinate system in which one of the three vectors representing the direction coincides with the unit vector in the z-axis direction, and the other two
It refers to the case where the signs of the xy components of the vector indicating one direction can be clearly defined.

For simplicity, a case where the relative relationship between the light source direction vectors is known will be described below. First, from the two solution matrices S ′ relating to the light source, only the calculation results of the light source whose relative relationship is known are taken out to form a matrix, and the determinant is calculated. Then, the matrix S'of the solution having the same code is selected by comparing with the code of the determinant obtained from the actual relative relationship. However, in the actual relative relationship, since only the sign of the xy component of the direction vector is known, it is necessary to determine the coordinate system so that the sign determines the sign of the determinant.

For all combinations of three direction vectors, the determinant code is the direction vector xy.
It has been confirmed separately that it can be determined only by the sign of the component. Of course, the matrix N'for the orientation of the surface is selected to be paired with the selected S '. Note that the same processing is performed when the relative relationship between the vectors of the normals of the three surfaces is known.

Next, with reference to FIGS. 5 and 6, the processing of the image matrix decomposition 2 in step SP3 will be described.
In the plurality of captured images, if the brightness of the light source of 6 or more images is constant or the ratio thereof is known, the process shown in FIG. 5 is performed, and the reflectance of 6 or more coordinate points is constant or Fig. 6 when the ratio of
Is processed.

First, the processing in the case where the brightness of the light source is constant or the ratio thereof is known according to FIG. 5 will be described. At step SP31, the temporary light source characteristic matrix S'obtained by the processing of decomposition 1 of the image matrix at step SP2.
From the above, only the light source characteristic vector s ′ of the image in which the brightness of the light source is constant or the ratio thereof is known is extracted. Then, when the brightness of the light source is constant, the following equation is constructed. However, the matrix A is the conversion matrix of (3,3) to be obtained.

[0039]

[Equation 6]

When the brightness ratio of the light source is known, instead of setting all 1s on the right side of the equation (6),
The values of these ratios are set respectively, but for simplicity, a fixed case will be described below. Using the symmetric matrix B = A ^T A of (3,3),

[0041]

[Equation 7]

It can be rewritten as Therefore, matrix B
Each element of is obtained by the linear least squares method, the obtained matrix B is subjected to singular value decomposition as follows, and the transformation matrix A is obtained.

[0043]

[Equation 8]

Subsequently, in step SP32, the true light source characteristic matrix S is calculated from the temporary light source characteristic matrix S'and the conversion matrix A.
Is calculated as follows.

[0045]

[Equation 9]

Further, in step SP33, the true surface characteristic matrix N is calculated from the temporary surface characteristic matrix N'and the inverse matrix A ^{-1 of the} conversion matrix as follows.

[0047]

[Equation 10]

Next, the processing when the reflectance is constant or the ratio thereof is known according to FIG. 6 will be described.
At step SP31 ′, from the temporary surface characteristic matrix N ′ obtained by the processing of decomposition 1 of the image matrix at step SP2,
Only the surface characteristic vector n'of the coordinate point whose reflectance is constant or whose ratio is known is extracted. Then, when the reflectance is constant, the following equation is constructed. However, the matrix A is the conversion matrix of (3,3) to be obtained.

[0049]

[Equation 11]

When the reflectance ratio is known, the values of the respective ratios are set instead of setting all 1s on the right side of the equation (12). The case will be described. This formula is (3,3)
Using the symmetric matrix B = AA ^T of

[0051]

[Equation 12]

It can be rewritten as Therefore, matrix B
Each element of is obtained by the linear least squares method, and the obtained matrix B is subjected to singular value decomposition in the following manner to obtain the transformation matrix A.

[0053]

[Equation 13]

Then, in step SP32 ', the true surface characteristic matrix N is calculated from the temporary surface characteristic matrix N'and the conversion matrix A.
Is calculated as follows.

N = N'A (16) Further, in step SP33 ', the true light source characteristic matrix S is calculated as follows from the temporary light source characteristic matrix S'and the inverse matrix A ^{-1 of the} conversion matrix. .

S = A ^-1 S '... (17) When the processing in either of these two cases ends, the processing in step SP3 ends, and the entire processing ends. If two or more plane directions and light source directions represented by the coordinate system of the television camera 4 are known, all the calculated relative plane directions and light source directions match the coordinate system of the camera 4. Can be converted into something absolute.

The embodiment described above is a case in which there is no shadow area in a plurality of picked-up images. Hereinafter, with reference to FIG. 7, a specific process of step SP2 when there is a shadow region in any one of a plurality of images of the cylinder 1 illuminated by different light source directions will be described. To do. However, step SP1 and step S
P3 performs the same processing.

First, in step SP21 ', an area on the image plane is set. However, the normal vectors of at least three points among the coordinate points of the set area should not be on the same plane. Next, in step SP22 ', the image data image-k that becomes a shadow even in a part of the set area is deleted from the image matrix I from the m captured images. However, the number of the plurality of image data forming the image matrix I obtained after deletion is m ′ (where m ≧
When m ′ ≧ 6), at least three of the light source direction vectors forming them are not on the same plane.

Then, in step SP23 ',
From the image matrix I of (n, m ′), only the image data of the coordinate points that do not always form a shadow and do not include the shadow area (f,
The image submatrix I ₁ of m ′) is constructed. However, n ≧ f ≧
In FIG. 6, it is assumed that the relative relationship between three light source direction vectors or normal vectors of planes that form this image sub-matrix I ₁ and are not on the same plane is known.

In step SP24 ', singular value decomposition is applied to the obtained image submatrix I ₁ . The processing method is step SP
It is exactly the same as 21. Then, in step SP25 ', two sets of temporary surface characteristic small matrix and light source characteristic matrix are calculated. The processing method is exactly the same as in step SP22. further,
One set is selected from the two sets of temporary surface characteristic small matrix and light source characteristic matrix obtained in step SP26 '. The processing method is exactly the same as in step SP23. At this stage, a temporary surface characteristic small matrix representing the characteristics of f surfaces and a temporary light source characteristic matrix representing m ′ light source characteristics are obtained.

Finally, in step SP27 ', from the obtained light source characteristic matrix representing the light source characteristics of m'and the image submatrix I _{2 of} (n-f, m') including the shadow area, n-
Compute for f unknown surface feature vectors. From the equation (2), I ₂ = N ₂ S (18), and the image submatrix I ₂ and the light source characteristic matrix S are known. Therefore, regarding the surface characteristic vector of each coordinate point, the linear least squares method is applied. If applied, it is possible to determine nf unknown surface characteristic vectors. However, if there are not three or more non-shadow image data relating to the surface characteristic vector to be obtained, that portion cannot be obtained. Finally,
A total of n surface characteristic matrices can be constructed by combining with f surface characteristic vectors already obtained. As described above, the specific process of step SP2 when any one of the plurality of captured images has a shadow area is ended.

[0062]

As described above, according to the present invention, a plurality of images are processed by capturing an image of an object whose shape illuminated by different unknown light source directions is unknown, and the image intensity data is obtained from each image. The image matrix is composed of, and the image matrix is decomposed into the product of the two sets of surface characteristic matrix and light source characteristic matrix using singular value decomposition, and one of these is selected, and the light source is selected from the processing results. The brightness of the light source and its direction are calculated in advance by calculating an appropriate conversion matrix according to the condition that the brightness is constant or the reflectance is constant or the ratio is known, and the conversion matrix is applied to the processing result. Even without giving, it is possible to accurately determine the relative surface orientation and reflectance of the three-dimensional object. Further, in terms of not giving the brightness and direction of the light source in advance, the relative brightness and direction of the light sources are extracted from a plurality of images of an object illuminated by different light source directions. It will be possible.

Further, in any of a plurality of images obtained by picking up an object illuminated by different light source directions, if any one of them has a shadow region, the above-mentioned decomposition by singular value decomposition and selection of the decomposition result are performed. In the process, as a pre-process, a region is set in the image plane coordinates, all image intensity data of the image in which the set region is a shadow is deleted from the image matrix, and the shadow is always cast from the resulting image matrix. The image small matrix that does not include the shadow area is constructed only with the image intensity data of the coordinate points that do not become, and as a post-processing, the image small matrix that contains the shadow area and the obtained relative light source brightness and direction are used. , By calculating a coordinate point whose relative reflectance or surface orientation is unknown, even if there is a shadow in a plurality of captured images, the relative surface orientation of the three-dimensional object or Reflectance It is possible to obtain a good time.

[Brief description of drawings]

FIG. 1 is a schematic block diagram of an embodiment of the present invention.

FIG. 2 is an overall flowchart showing an object and light source information extraction method according to an embodiment of the present invention.

FIG. 3 is a step SP1 of the overall flow chart shown in FIG.
It is a flow chart showing the details of.

FIG. 4 is a step SP2 of the entire flow chart shown in FIG.
It is a flow chart showing the details of.

5 is a step SP3 of the overall flow chart shown in FIG.
It is a flow chart showing the details of.

FIG. 6 is a step SP3 of the overall flow chart shown in FIG.
It is a flow chart showing the details of.

FIG. 7 is a step SP of the overall flow chart shown in FIG. 2 when a shadow region exists in a plurality of captured images.
It is a flowchart which shows the detail of 2.

FIG. 8 is a diagram for explaining the principle of a conventional object information extraction method.

FIG. 9 is a diagram for explaining the principle of a conventional object information extraction method.

FIG. 10 is a diagram for explaining a conventional object information extraction method.

[Explanation of symbols]

 1 Target Object 2 Light Source 3 Moving Device 4 Television Camera 5 Image Processing Device 6 CPU

Claims (3)

[Claims] 1. Processing a plurality of images of an object whose shape illuminated by different unknown light source directions is unknown,
An object and a light source information extraction method for extracting the relative reflectance and surface orientation of the target object and the relative brightness and light source direction of the light source, wherein an image matrix composed of the image intensity data is obtained from each of the images. The first step of configuring and the image matrix configured in the first step are decomposed into a product of two sets of surface characteristic matrix and light source characteristic matrix using singular value decomposition, and the one set is selected. From the second step and the processing result of the second step, a predetermined conversion matrix is calculated under the condition that the brightness of the light source is constant or the reflectance is constant or the ratio thereof is known, and the conversion is performed. By applying a matrix to the processing result of the second step, a third step of extracting the relative reflectance and surface orientation of the object and the relative brightness and light source direction of the light source, Light source information Way out. 2. The second step, when a shadow area is present in any one of the images, as a pre-processing, a step of setting an area in image plane coordinates, and the set area. A step of deleting from the image matrix all image intensity data of an image having a shadow, and an image sub-matrix not including a shadow region is formed only from the image intensity data of coordinate points that are not always a shadow from the image matrix. As a post-processing, the image submatrix including the shadow area obtained by removing the image submatrix not including the shadow area from the image matrix and the obtained relative lightness and direction of the light source are 2. The object and light source information extraction method according to claim 1, including the step of calculating a coordinate point whose unknown reflectance or surface orientation is unknown. 3. Processing a plurality of images of an object whose shape illuminated by different unknown light source directions is unknown,
An object and a light source information extraction device for extracting the relative reflectance and surface orientation of the object and the relative brightness and light source direction of the light source, the imaging means for capturing an image of the object, the object Light source moving means for changing the relative positional relationship between the light source and the light source, and an image matrix composed of image intensity data from images from different light source directions picked up by the image pick-up means by using singular value decomposition. Condition that the brightness of the light source is constant or the reflectance is constant or its ratio is known from the processing result after decomposing into a product of the characteristic matrix and the light source characteristic matrix and selecting one set An image processing for extracting a relative reflectance and a surface orientation of the object and a relative light source brightness and a light source direction by calculating a predetermined transformation matrix with the processing result and applying the transformation matrix to the processing result. Comprising means, the object and the light source information extracting device.