JPH0792369B2 - Image measuring device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image measuring device,
In particular, the present invention relates to an image measuring device for recognizing and measuring a three-dimensional position of an object, which is used for recognizing and measuring the motion and posture of a human body or machine.

[0002]

2. Description of the Related Art Conventionally, as an object measuring method using an image, an active measuring method using light or sound wave projection, a measuring method using constraint of projection geometry, a measuring method using stereo matching, etc. are known. Among them, the active measurement method using light or sound wave projection is a method of irradiating a laser beam or the like and obtaining the distance of an object from its response time or a reflection pattern, but it has a drawback that there are restrictions on the usage environment and the target. Further, the measurement method based on the constraint of the projection geometry is a method of estimating the shape and position of the target from the surface and the ridge of the target object by using the geometric constraint based on the projection condition. But this method
This method is effective only for artifacts whose surface and ridge can be clearly measured. The measurement method by stereo matching is
Although it is a method using a compound eye, searching for corresponding points in pixel units is an issue for measuring the position of an object.

Further, as an attempt to solve these problems, an apparatus as disclosed in Japanese Patent Application No. 63-199163, "Image Measuring Apparatus", has been proposed.

FIG. 11 is a diagram showing the configuration of the above-mentioned image measuring device, and FIGS. 12 and 13 are diagrams for explaining the operation of FIG.

In FIG. 11, picked-up images of an object picked up at different positions from the first and second image pickup devices 101 and 102 are given to contour image extraction devices 103 and 104, where the contour image data of the object is obtained. can get. Each contour image data is given to the linear approximation devices 105 and 106, and
As shown in FIG. 2, a straight line list of straight line approximation data at the multiresolution of the contour image data is obtained. Here, the object is treated by a model of a set of cones and cylinders as shown in FIG. The parallel line segment pair extraction devices 107 and 108 extract from a combination of parallel line segment pair candidates and straight line lists obtained on the image from the cone and the cylinder. Target axis computing device 10
Reference numerals 9 and 110 calculate the target axes of the parallel line segment pairs, and the straight line calculation device 111 matches the target axes obtained from the respective images as shown in FIG. 13 to extract the cone and the cylinder. Obtain straight line data showing the position of the axis in three dimensions.

However, the above method also has the following problems in its practical application. First, the linear approximation processing of the critical line by multi-resolution is performed on all contour lines, which is inefficient. The process of searching for a cone with each combination from the obtained set of approximate straight lines becomes enormous when the contour shape is complicated and the number of elements of the set is large, which is not practical. Furthermore, when the processing in the time axis direction is not taken into consideration, for example, when recognizing / measuring a person's motion or the like, it is necessary to perform the above-described processing on a moving image, which requires a huge amount of processing and is practical Not. Also, 2
Not applicable when two cones overlap in the image.
Therefore, it cannot be applied to applications such as measuring the postures of limbs of the entire human body.

[0007]

The above-mentioned three methods are as follows.
Although they have their own merits, they have their respective adaptation ranges. For example, in the application of recognizing and measuring humans, animals, etc., the active measurement method limits the environment and the object,
The method using the projected geometric constraint has a problem that it is difficult to extract features such as faces and edges, and the stereo method has difficulty in finding corresponding points. Also, in the last-mentioned example, there is a problem that the arithmetic processing is inefficient and it is not possible to deal with overlapping of cones.

Therefore, the main object of the present invention is that it is adaptable to natural objects such as the human body and is not easily affected by partial hiding or overlapping regardless of the active retrieval method, and is efficient for moving images. The present invention is to provide an image measuring device that can be processed in a physical manner and can measure the state of motion and posture of an object.

[0009]

An invention according to claim 1 is an image measuring apparatus for measuring an image of an object, wherein first and second image pickup means are provided at at least different viewpoints, respectively. Three-dimensional position detecting means for detecting the three-dimensional position of the feature point in the object from the output images of the first and second imaging means, and a motion model for storing data on each motion and posture of the motion model of the object. From the data storage means and the three-dimensional position of the feature point in the object obtained by the three-dimensional position detection means or its temporal change, the movement of the object that emits the three-dimensional position of the similar feature point or its temporal change And posture data from the data stored in the motion model data storage means, and the direction of the object is estimated from the retrieved motion and posture data and the detected three-dimensional position. For each of the retrieved motion and posture data, the three-dimensional position of one of the feature points obtained by the three-dimensional position detecting means is calculated based on the estimated direction of the object. As a reference, the three-dimensional position of another feature point or the area of the object obtained by the area extraction means is estimated, and this is also obtained by the three-dimensional position or area extraction means of the feature point actually obtained by the three-dimensional position detection means. Based on the difference with the area of the target object, the degree of coincidence is evaluated, the best match among the searched motions and postures is selected, and the motion and posture data and the direction of the target object are selected. The estimation error analysis and verification means for correcting and outputting as a measurement result of the motion and orientation of the object is configured.

More preferably, it includes an object area extracting means for extracting an area of an image of the object from the output images of the first and second image pickup means, and the three-dimensional position detecting means is extracted by the object area extracting means. The three-dimensional position of the feature point is detected from the region of the target object image, and based on the motion and posture recognized by the motion recognition unit, the motion of the target object based on the shape of the target region obtained from the target region extraction unit Obtain attitude data.

[0011]

Further, the three-dimensional position detecting means includes first and second three-dimensional position detecting means for extracting an area of a specific portion of the area of the object image extracted by the object area extracting means. Based on the output image of the image pickup means and the representative point in the area of the specific portion extracted by the object partial area extracting means, the three-dimensional position of the feature point in the object corresponding to the representative point is detected.

Further, the motion retrieval means includes a parameter extraction means for extracting a characteristic parameter independent of the three-dimensional position of the characteristic point and its temporal change to a free reference position and direction of the three-dimensional position of the object. Similar feature parameters for the exercise model data stored in the model data storage means are stored, and a search is performed based on the similarity between the two feature parameters.

[0014]

In the image measuring device according to the present invention, the image of the object is taken by the first and second image pickup means from different viewpoints, and the three-dimensional position of the feature point in the object is detected from the output image. , A motion model that stores movement and posture data of an object that emits a three-dimensional position of a similar feature point or its temporal change from the detected three-dimensional position of the feature point in the object or its temporal change The direction of the object is estimated from the retrieved motion and orientation data and the detected three-dimensional position based on the estimated direction of the object for each retrieved motion and orientation data. , The three-dimensional position of another feature point or the area of the object is estimated with reference to one three-dimensional position of the detected feature points, and the three-dimensional position of the feature point or the object actually obtained Territory of Based on the difference between the motions and postures, the degree of matching is evaluated, the one that most matches the searched motions and postures is selected, and the data of the motions and postures and the direction of the target are corrected, and the target It is output as the measurement result of the movement and posture of the object.

[0015]

1 is a schematic block diagram of an embodiment of the present invention. First, with reference to FIG. 1, a schematic configuration of an embodiment of the present invention will be described. The image pickup means 1 and 2 are arranged at different positions and pick up an image of a person as an object. The object area extracting means 3 and 4 extract the area of the person of the object from the image data obtained from the imaging means 1 and 2. The skin color region extracting means 5 and 6 extract skin color regions from the impurity regions of the object. Three-dimensional position calculation means 7
Calculates the position of the head from the area of the person obtained by the object area extracting means 3 and 4, and then calculates the three-dimensional position of the head based on the triangulation method. Similarly, after the face and hand positions are calculated from the skin color regions obtained by the skin color region extracting means 5 and 6, the face and hand positions are calculated based on the triangulation method.
Calculate the three-dimensional position of the hand. The parameter extraction means 8
From the obtained three-dimensional position data of each point, parameters independent of the direction in which the person is facing and the horizontal position of the person are extracted. The motion search means 9 searches the data stored in the exercise model database 10 for a motion and state that generate a parameter similar to the parameter calculated by the parameter calculation means 8 and outputs it.

The direction estimating means 11 estimates the direction in which the person is facing. The estimation error analysis verification means 12 uses the three-dimensional position of one of the feature points obtained by the three-dimensional position calculating means 7 as a reference, based on the direction of the person estimated by the direction estimating means 11, and determines the other features. The three-dimensional position of the point or the three-dimensional position of the feature point obtained by the three-dimensional position calculating means 7 is estimated by estimating the three-dimensional position of the object Based on the difference between the object and the area obtained by the area extracting means 3 and 4, the degree of coincidence between them is evaluated, and the most matching one of the retrieved movements and postures is selected, and the movement and posture are selected. And the direction of the object are corrected and output as the measurement result of the motion and orientation of the object.

FIG. 2 is a schematic diagram for explaining the installation state of the image pickup apparatus shown in FIG. In FIG. 2, the imaging devices 1 and 2 are 120 cm above the floor, respectively, and are at a distance of 80 c
They are installed at intervals of m, and their optical axis directions are horizontal and parallel to each other. In such a state, the motion of the person 20 who is about 2 to 6 m away from the imaging means 1 and 2 is imaged. Further, in FIG. 2, the three-dimensional coordinate axes are indicated by X, Y, Z.

FIG. 3 is a view for explaining an example of area extraction by the object area extraction means shown in FIG. 1, and FIG. 4 is a view for explaining the operation of the three-dimensional position detection means shown in FIG. Yes, FIG. 5 is a graph for explaining an example of position detection by the three-dimensional position detecting means, FIG. 6 is a transition diagram of the motion model of the motion model database, and FIGS. 7 and 8 show the operation of the motion search means. 9 is a graph for explaining FIG.
Is a graph for explaining an example of the direction estimation by the direction estimation means shown in FIG. 1, and FIG. 10 is a diagram for explaining the operation of the estimation error analysis verification means.

Next, the specific operation of the embodiment of the present invention will be described with reference to FIGS. A person image 20 as an object is picked up by the image pickup means 1 and 2 in the state shown in FIG. 2, and image data 31, 3 as shown in FIG.
2 is provided to the object area extracting means 3 and 4. The object area extracting means 3 and 4 are the image data 31 and 32 shown in FIG.
The area 33 of the person of the object is extracted from. Further, the skin color area extracting means 5 and 6 extract the skin color area 34 from the area 33 of the target person. Here, the flesh-colored region 34 in the region of the person is a candidate for a portion of the human body such as a hand or face where the skin is exposed.

Next, the three-dimensional position calculating means 7 extracts candidate points 41 of the head from the human region 33 obtained from the image data 31 and 32, respectively, as shown in FIG. Based on the triangulation formula, the three-dimensional position v0 of the head
To calculate. 5A and 5B show an example of the locus of the three-dimensional position v0 of the head obtained by the three-dimensional position calculating means 7, and in particular, (a) shows the XY direction,
(B) is plotted in the Z-X direction. next,
As shown in FIG. 4B, the three-dimensional position calculating means 7 is
After extracting the candidate points of the hand 42, the three-dimensional positions v0 ', v1', V2 'of each point are calculated based on the triangulation method in the same manner as the head position. A point closer to the head position v0 is defined as a face position v0 ″, and the heading position from other points is assumed to be the front of the person, and a point to the right of the person is a right hand position v1. , The other is output as the left-hand position v2.

The parameter extracting means 8 extracts, from the three-dimensional position data of each point obtained from the three-dimensional position calculating means 7, independent parameters for the direction in which the person is facing and the horizontal position of the person. . Here, the height P1 of the head, the moving speed P2 of the head in the horizontal direction, the moving speed P3 of the head in the vertical direction, the distance P4 between the head and the right hand, and the distance P5 between the head and the left hand. Extract five parameters. The motion model database 10 is a state (sta) at each motion and each time for a typical motion model of a person.
te) and the data is stored in advance.

FIG. 6 shows a typical transition of the whole body motion in the room of a person, and Table 1 shows the classification of each motion and the state at each time.

[0023]

[Table 1]

The motion model database 10 has a state at each motion and each time.
For each of these, the data of the joint angles of the human body and the data of the moving speed of the head in the horizontal direction in which the person is facing are stored as shown in Table 2.

[0025]

[Table 2]

The motion retrieving means 9 has the same parameters as those calculated by the parameter calculating means 8 which are generated from the data of the motion model database 10 in advance when each motion transitions based on the transition diagram of FIG. Is calculated and stored in advance, and when a parameter is actually input from the parameter extraction means 8, an operation (motion) and a state (st) where a parameter similar to that is generated.
ate) and retrieve and output.

The operation of the operation search means 9 will be described in more detail with reference to FIGS. 7 and 8. Figure 7
7 (a) and 7 (b) show the elements that each operation (motion) and state (state) transit by their code numbers, and FIGS. 7 (c) and 7 (d) show the parameters generated at that time. Is calculated. In FIG. 7, the horizontal axis represents time, and the unit is called a frame. Here, one frame is 0.1 second. The code numbers i and j representing the respective states of the respective operations are expressed by vertical vectors X which are orthogonal to each other.
It is represented by (i, j) and is called a motion vector.

The values of the respective parameters pn (n: 1 to 5) are quantized into eight states, which are represented by vertical vectors Pn having eight elements and orthogonal to each other. A vertical vector P having 40 elements is obtained by combining the vectors Pn representing the five parameters.
Represents all parameters, and this is called a parameter vector.

As shown in FIGS. 7A and 7B, time t
The relationship between the motion vector Xt and the parameter vector Pt is stored in the associative memory matrix Mk for each parameter generated when each state of each motion transits in accordance with the above. Then k
Represents the time difference from the parameter that occurred when the action occurred.

[0030]

[Equation 1]

Next, the operation when parameters are actually input from the parameter calculation means 8 will be described with reference to FIG. 8A and 8B are examples of parameters input from the parameter calculating means 8. Each of the parameters pn 'is represented by a vertical vector Pn' in the same manner as above.
However, as shown in FIG. 8B, since the position of the hand cannot be detected due to hiding or the like, the corresponding parameter p
There are cases where 4 and p5 cannot be obtained, in which case they are represented by the 0 vector. In the same manner as described above, the vector Pn ′ representing each parameter is combined to obtain the parameter vector P ′ having 40 elements. Using this parameter vector P'and the associative memory matrix Mk, the recollection vector S is obtained as shown in the equation (2). This recollection vector S is easily orthogonal to each other and each motion vector X
It can be decomposed into (i, j). Since the coefficient a indicates the correlation value between the parameter vector P stored in advance and the input parameter vector P ′, the motion vector X (i, j) where the value of the coefficient a is large is three from the largest. The extracted code number (i, j) and the coefficient a are output to the direction estimation means 11 and the estimation error analysis means 12.

[0032]

[Equation 2]

FIGS. 8 (c) and 8 (d) show FIGS. 8 (a) and 8 (b).
The code number (i, j) at which the coefficient a is maximized is shown for the input example of the parameter shown in (4), and as shown in FIG. It can be seen that the transition can be estimated. Moreover, even if some parameters at a certain time cannot be input, it can be understood that the motion can be estimated from the parameters at another time and other parameters.

When estimating the motion at a certain time as shown by the value of k in the equations (1) and (2), stable motion estimation is performed by using the parameters from four frames before to four frames later. I am doing it. Even when performing continuous motion estimation while measuring continuous motions of a person, stable motion estimation can be performed by performing motion recognition at a certain point after the parameters after four frames are input.

The direction estimating means 11 is capable of transitioning with respect to two code numbers searched four frames before and four frames after assuming one of the three code numbers searched at a certain time. A series (i, j) k ', and the horizontal movement of the head obtained by the three-dimensional position calculating means 7 based on the extracted series (i, j) k'and each (i shown in Table 2 stored in the motion model database 10). , J), the direction φ in which the person is facing is estimated by the equation (3) from the horizontal head velocity data sm.

[0036]

[Equation 3]

The direction φ of the person when the three code numbers retrieved at a certain point are assumed respectively.
Is output to the estimation error analysis verification means 12.

FIG. 10 shows the horizontal (X) of the head shown in FIG. 5B based on the operation and state shown in FIGS. 9C and 9D.
The direction of the person estimated from the movement in the (-Z) direction is shown. The estimation error analysis verification means 12 corresponds to the estimated value φ of the direction of the person obtained from the direction estimation means 11 and the searched motion and state when the three code numbers searched at a certain time are respectively assumed. Data of each joint angle of the human body is input from the motion model database 10. Further, the position information of each point obtained from the three-dimensional position calculating means 7 is input. Then, as shown in FIG. 10 (a), the direction φ of the input person based on the head position v 0,
The posture of the human body in three dimensions is reconstructed from each joint angle data Θ (θ0 to θ1). The three-dimensional position V '(v0' to vm ') of each reconstructed human body part is calculated according to the equation (4).

[0039]

[Equation 4]

For the left and right hand positions v1 and v2 obtained by the three-dimensional position calculating means 7, the three-dimensional position calculating means 7
Since the left and right hands may be mistaken, if the difference between the positions v1 'and v2' obtained by the equation (4) is compared and the left and right are replaced, the sum of the squares of the difference distances is small. , Replace the left and right. Then, based on the equation (4), v
The direction φ of the person and the joint angles Θ are corrected to φ ′ and Θ ′, respectively, so that 1 ′ and v2 ′ match v1 and v2.
At this time, since the correction method is not uniquely determined, the correction is made so that the horizontal error evaluation function H shown in the expression (5) becomes the minimum. The coefficients c and d in the equation (5) are for evaluating the certainty of the estimated motion based on the deviation from the model data such as the joint angle, and their values are different for each motion.

[0041]

[Equation 5]

Further, as shown in FIG. 10B, the first position Xmin (Y) at which a region appears on the horizontal scanning line Y from the region of the person obtained from the object region extracting means 3 and 4.
And the final position Xmax (Y) is obtained. Also, the fourth (4)
The position of each image group obtained from the position (xc, yc, zc) of the image pickup device from the position V ′ of each point of the human body obtained by the formula is reconstructed from the position so that it can be obtained as in the formula (6). Area Xmin ′ (y) of the person caused by the human body posture,
Xmax ′ (Y) is obtained as shown in the equation (7), and the direction of the person is calculated so that the value of the estimation error evaluation function H ′ that exceeds the sum of the differences between the two shown in the equation (8) becomes the minimum. φ ′ and each joint angle Θ ′ are corrected to φ ″ and θ ″, respectively.

[0043]

[Equation 6]

Of the three code numbers retrieved at a certain time point, the one having the smallest value of the estimation error evaluation function H'is selected, and the code number (i, j) indicating the operation and state thereof is selected.
Then, the direction φ ″ of the person, each joint angle Θ ″, and the three-dimensional position v0 of the head are output as the recognition measurement result of the motion posture of the person at that time.

With the configuration as described above, in this embodiment, the motion of the person can be efficiently estimated by searching the motion model database 10 from the parameters that can be extracted relatively easily from the image. Based on this, the contour line of the human region can be efficiently analyzed, and the actual motion deviated from the motion model stored in the database can be recognized stably and efficiently. For example, in the conventional example, the position cannot be measured unless both of the two straight lines generated from the portion modeled by each cone are extracted as contour lines, but according to one embodiment of the present invention, one can be obtained. If so, the position can be measured stably.
Further, in the conventional example, the contour line must be approximated to a straight line at multiple resolutions, but this is not necessary in the embodiment of the present invention,
It can be measured efficiently.

In the above-described embodiment, after extracting the area of the object, the partial area of the object is extracted using the color information to increase the information amount of the characteristic parameter for searching the motion model database 10. However, this part can be easily simplified.

Further, in the above-described embodiment, the estimation error analysis / verification means 12 uses both the position of each point obtained from the three-dimensional position calculation means 7 and the person area obtained by the object area extraction means 3 and 4. Although the analysis of the estimation error and the evaluation and verification are performed by using, either one of them may be simplified.

[0048]

As described above, according to the present invention, the movement and posture of an object such as a person can be efficiently estimated by detecting a motion model database from several feature points extracted from a video. By efficiently analyzing the contour line of the object area based on this estimation result,
It is possible to stably and efficiently recognize the actual motion deviated from the motion model stored in the database. Therefore, it becomes possible to recognize the posture and motion of the human body and the like and measure the position using the image, which has been difficult in the past. Moreover, in the present invention, by utilizing the knowledge of the motion model about the object, it can be applied to a target such as a living thing or a machine that can acquire the knowledge of the motion model in advance regardless of human beings, etc. Wide and practically effective.

[Brief description of drawings]

FIG. 1 is a schematic block diagram of an image measuring device according to an embodiment of the present invention.

FIG. 2 is a schematic diagram for explaining an installation situation of the image pickup apparatus shown in FIG.

FIG. 3 is a diagram for explaining an example of region extraction by the region extracting means shown in FIG.

FIG. 4 is a diagram for explaining the operation of the three-dimensional position calculating means shown in FIG.

FIG. 5 is a diagram showing a graph for explaining an example of position detection by a three-dimensional position calculating means.

6 is a transition diagram of a motion model of the motion model database shown in FIG.

FIG. 7 is a diagram showing a graph for explaining the operation of the operation search means shown in FIG.

FIG. 8 is a diagram showing a graph for explaining the operation of the operation search means.

FIG. 9 is a diagram showing a graph for explaining an example of direction estimation by the direction estimating means shown in FIG.

FIG. 10 is a diagram for explaining the operation of the estimation error analysis verification means shown in FIG.

FIG. 11 is a schematic block diagram of a conventional image search device.

FIG. 12 is a diagram showing contour image data of an object obtained from an imaging device in a conventional image measuring device.

FIG. 13 is a diagram showing an example of modeling an object in a conventional image measuring device.

[Explanation of symbols]

 1, 2 Imaging means 3, 4 Target area extraction means 5, 6 Skin color area extraction means 7 Three-dimensional position calculation means 8 Parameter calculation means 9 Motion retrieval means 10 Motion model database 11 Direction estimation means 12 Estimation error analysis verification means 31, 32 image data 33 human area 34 flesh color area 41 head candidate 42 face and fingertip candidate

Continuation of front page (51) Int.Cl. ⁶ Identification number Office reference number FI technical display location G06T 7/20

Claims (4)

[Claims] 1. An image measuring device for measuring an image of an object, comprising: first and second image pickup means provided respectively at at least different viewpoints; and output images of the first and second image pickup means. Three-dimensional position detecting means for detecting the three-dimensional position of a feature point in the object, motion model data storage means for storing data relating to each motion and posture of the motion model of the object, obtained from the three-dimensional position detecting means From the three-dimensional position of the feature point in the object or its temporal change, the motion model data storage means stores the data of the movement and posture of the object that emits the three-dimensional position of the similar feature point or its temporal change. Motion searching means for searching from the stored data, motion and posture data searched by the motion searching means and three-dimensional detected by the three-dimensional position detecting means Direction estimating means for estimating the direction of the object from the position, and for each motion and posture data searched by the motion searching means, based on the direction of the object estimated by the direction estimating means, the three-dimensional position The one-dimensional position of the feature points obtained by the detection means is used as a reference to estimate the three-dimensional position of another feature point or the area of the object obtained by the area extraction means, and the three-dimensional position is actually estimated. Based on the difference between the three-dimensional position of the feature point obtained by the original position detection means or the area of the object obtained by the area extraction means, the degree of coincidence between them is evaluated, and among the searched movements and postures Estimated error analysis and verification means is provided for selecting the one that most matches, correcting the motion and orientation data and the direction of the object, and outputting the result as the measurement result of the operation and attitude of the object. Image measurement device. 2. The apparatus further comprises a target area extraction means for extracting a target image area from the output images of the first and second image pickup means, wherein the three-dimensional position detecting means extracts the target area. The three-dimensional position of the feature point is detected from the area of the object image extracted by the means, and the motion recognition means, based on the recognized motion and posture, the object area obtained from the object area extraction means. 2. The image measuring device according to claim 1, wherein the motion and orientation data of the object is obtained from the shape of the object. 3. A target part partial area extracting means for extracting a specific part of the area of the target object image extracted by the target object area extracting means, the three-dimensional position detecting means comprising: Based on the output images of the first and second image pickup means and the representative points in the area of the specific portion extracted by the object partial area extraction means,
The image measuring device according to claim 2, wherein the three-dimensional position of the feature point in the object corresponding to the representative point is detected. 4. A parameter calculation for calculating an independent feature parameter in a free three-dimensional reference position and direction of an object from the three-dimensional position and temporal change of the feature point obtained from the three-dimensional position detecting means. The motion search means stores characteristic parameters similar to the exercise model data stored in advance in the exercise model data storage means, and stores in advance the parameters calculated by the parameter calculation means. The image measuring device according to claim 1 or 3, wherein the search is performed based on a degree of similarity with a feature parameter that is present.