JP3276010B2 - How to enter information - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention directly captures the movement of a pointing motion object such as an operator's limbs and a pointing device as an image,
The present invention relates to an information input method for performing image processing and inputting information to a pointing target object or the like, which is expressed by the movement of the pointing motion object, to an information processing apparatus.

[0002]

2. Description of the Related Art At present, a keyboard and a mouse are generally used as input devices of an information processing apparatus (computer). However, the use of these devices requires a certain degree of skill and is not something that everyone can use immediately. At present, many people say that "computers are difficult" because of such poor interfaces. Originally, conversations with computers should be conducted using a natural method similar to that of conversation between humans. In this regard, the interface using "look and feel" can be easily used by anyone without skill.
It is thought to greatly contribute to the realization of “a computer that can be used by anyone”. Japanese Patent Application No. 3-053565, "Information Input Method" and Fukuma, Mase, Suenaga "Experimental System for Direction Detection Using Image Processing" (IEICE 1991)
Spring National Convention Lecture Paper A-251) partially solves the above problem, detects the position of the fingertip or the indicator from the image input from the TV camera, and identifies the pointing target object and recognizes the command operation. It is what you do.

[0003]

The above-mentioned prior art has the following disadvantages.

(1) In this paper, a description of a method for recognizing the presence or absence of a fingertip from the tip point of the fingertip is simply described as "based on the thickness and length of the finger". Not explicitly stated. Therefore, there is a possibility that the pointing action object may be erroneously detected due to an unnecessary object or noise on the screen.

[0005] (2) Regarding a method of determining a reference point, the publication states that "a point in a three-dimensional space where a straight line group obtained by calibration intersects is set as a reference point". "In the case of performing an instructing action, this straight line group easily intersects almost one point as long as the arm is moved without changing the upper body," but in reality,
The straight line groups are often at “twisted positions” in space with each other, and do not intersect at one point. Furthermore, no algorithm for calculating the reference point in such a case is described.

A first object of the present invention is to provide an information input method for strictly determining a pointing motion object.

A second object of the present invention is to provide a method for inputting information for correctly determining a reference point when a plurality of straight lines indicating a designated direction do not intersect at one point in space.

[0008]

A first information input method according to the present invention processes a plurality of imaging signals obtained from a plurality of imaging devices arranged so as to have different fields of view with respect to a pointing motion object. Then, obtain the three-dimensional information of the pointing action object, input this to the information processing device, and on a straight line connecting a tip point obtained from a specific portion of the pointing action object and a predetermined reference point. Determining the pointing target object present;
This is input to the information processing apparatus, in the information input method, the point closest to the pointing target object, a point considered as a part of the pointing operation object as the tip point candidate, for the tip point candidate, On the imaging signal, a search is performed on the basis of the tip point candidate, and from the search result and the feature amount of the pointing action object itself, the tip point candidate corresponds to a part of the pointing action object and becomes a tip point. It is determined whether or not it can be obtained.

According to a second information input method of the present invention, a plurality of image pickup signals obtained from a plurality of image pickup devices arranged so as to have different fields of view with respect to a pointing action object are processed, and the pointing action object is processed. The three-dimensional information is obtained and input to the information processing apparatus, and the pointing target object existing on a straight line connecting a tip point obtained from a specific portion of the pointing motion object and a predetermined reference point And inputting the same to the information processing apparatus. In the information input method, the pointing target object indicates a plurality of the known pointing target objects, thereby passing through the pointing target objects and the pointing target object. A straight line is determined, and the reference point is determined based on a sphere having the smallest radius among spheres passing through the center, with a certain point in the three-dimensional space near the convergence of these straight lines. I do.

[0010]

By checking whether the extracted tip point really belongs to the pointing motion object, it is possible to prevent an unnecessary object on the screen or erroneous detection of the pointing motion object due to noise. Also, when a plurality of straight lines indicating the designated direction do not intersect at one point in space, the straight line group correctly determines the reference point based on the sphere having the closest radius among the spheres passing through the inside. A reference point can be determined.

[0011]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of the present invention, FIG. 2 is an explanatory diagram showing a verification method for a candidate for a tip point, and FIG.
FIG. 4 is an explanatory diagram showing a reference point search algorithm, and FIG. 4 is a diagram showing an example of a method of determining a reference point search range.

In this embodiment, a command for the pointing target object 5 projected on the screen 4 is specified by the hand 6 (pointing operation object) of the operator 10, and the wall of the hand 6 has a different field of view. And two imaging signals obtained by the two television cameras 1 and 2 (imaging devices) arranged on the ceiling, respectively, are processed by the information input device 3 to obtain the command and the pointing target object 5, and these are processed by information processing. This is input to the device 9. The coordinate system of the three-dimensional space is set as shown in FIG. 1 with the origin O at the lower right corner of the screen 4.

Next, the operation of this embodiment will be described.

First, the two TV cameras 1, 2 and 6
Is input to the information input device 3 and processed by the device 3 to obtain a three-dimensional space Ox of the tip 7 of the hand 6.
Determine the coordinate position in yz. Next, from the tip point 7 and the reference point 8, the designated target object 5 is placed in the aforementioned Japanese Patent Application No. 3-05356.
Determined using the method described in the specification of No. 5. Next, hand 6
Is identified, and predetermined information corresponding to the state or the state sequence is indicated by the target object 5.
Is input to the information processing apparatus 9 as command information for the above.

Here, a method of determining the tip point 7 is shown in FIG.
This will be described below. Normally, when a human performs a pointing operation using a hand or an indicator, it can be considered that the tip of the hand or the indicator is located closest to the pointing target object. In the aforementioned Japanese Patent Application No. Hei 3-053565, the coordinates of the tip point 7 are as follows:
The coordinates of the position closest to the pointing target object 5 among the pointing motion objects are determined. However, in this method, when a background object, noise in an image, or the like exists closer to the screen than the pointing operation object, there is a risk of detecting an incorrect tip point. Therefore, in the present embodiment, the binarized continuous area 11 is scanned in the direction of the screen 4 to extract a tip point candidate, and then the feature amount of the hand 6 as the pointing action object (here, the finger Using the thickness and the length up to the first joint), it is determined whether or not the tip point candidate is the pointing motion object. This algorithm will be described with reference to FIG.

(1) The hand 6, which is the pointing motion object, is cut out from the background, and a binarization process is performed to obtain a binarized hand 11.

(2) Search for a continuous area 11 in the direction of the screen 4 in the image toward the screen 4.

(3) The tip of the continuous area 11 closest to the screen 4 is set as a tip point candidate 12.

(4) A circle 23 having the radius of the length 20 to the first joint of the forefinger around the tip point candidate 12 is drawn.

(5) Scan the circumference of the circle 23 and measure the length 21 across the continuous area 11.

(6) The thickness 22 and the length 21 of the index finger are compared, and if the difference between the two falls within a certain range, the tip point candidate 12 is set as the tip point 7.

(7) Otherwise, the tip point candidate 12 is discarded.

Here, the length of the forefinger up to the first joint is used as the feature quantity, in order to stably detect the tip point regardless of the angle of the forefinger. Further, when the tip point candidate is discarded, another search can be performed by searching again. In this case, a candidate point (group) is extracted with a restriction of being “convex in the screen direction”, and a search is performed in the order of “closer to the screen”, so that the tip point is correctly detected from background objects and noise. It is possible to do.

Next, a method of determining the reference point 8 will be described with reference to FIG. In the above publication, a known point group on a screen is designated, and a reference point is obtained as an intersection of a designated line at that time. However, it is rare that the designated straight lines thus obtained intersect at one point in the three-dimensional space. The publication only states that "it is easily known that it almost intersects at one point", but mentions how to find a reference point when a straight line does not intersect at one point. FIG. 3 shows an example of an algorithm for obtaining a reference point in such a case.

(1) The pointing target object 14 is displayed at the center of the screen 4 and a pointing operation is performed.

(2) The tip point 16 is determined, and one straight line 17 in the three-dimensional space is determined from the position (known) of the pointing target object 14.

[0028] (3) at the four corners of the screen to display the instruction target object 15 _{1-15 4} sequentially, linear 1 in the same way
To determine the 8 _{1-18 4.}

(4) When a sphere 32 having a radius 31 centered on a point 30 in a three-dimensional space is drawn, all the straight lines 17,
18 _{1-18 4} Out of the sphere that passes through the inside radius 31 is the reference point 8 the center point of the smallest sphere.

Here, if the calculation of the radius 31 is performed for all the points in the three-dimensional space, the calculation amount becomes enormous. Therefore, the calculation is performed only for the space near the convergence point. An example of a method of calculating the space near the convergence point will be described with reference to FIG. Here, Japanese Patent Application No. 3-05
It is assumed that three straight lines 40, 41, and 42 in a three-dimensional space are determined by the correction operation described in the specification of Japanese Patent No. 3565.

(1) The closest point 4 between the two straight lines 40 and 42
Find 3,44. This can be uniquely derived from a straight line equation.

(2) Similarly, for all combinations of two straight lines, the closest approach point (46a, 46a in FIG. 4)
b, 47a, 47b).

(3) The coordinates of all these closest points are searched, the minimum value and the maximum value are obtained for each coordinate axis, and the reference point search range (Xrange, Yrange in FIG. 4) is determined between them.
(The z axis is omitted)).

(4) For a rectangular parallelepiped (Xrange * Yrange * Zrange) in a three-dimensional space surrounded by the search range of each axis, the radius 31 of the sphere is obtained.

In the actual calculation, if the spatial resolution is set to about 1 mm, sufficiently practical accuracy can be obtained, and the calculation time can be sufficiently reduced. When a point on the screen 4 is designated, the four corners and the center point of the screen 4 are designated. In this case, a straight line obtained by designating the center point may deteriorate the accuracy of the convergence point. is there. In such a case, except for the straight line by the center point, the reference point 8
Make a decision.

In this embodiment, the human hand 6 is used as the pointing operation object, but it may be another body part such as a foot or a mouth or an indicator such as a stick. further,
In the present embodiment, the two television cameras 1 and 2 are arranged on the wall surface and the ceiling, but it is sufficient that a plurality of television cameras are arranged at positions where stereo images can be obtained.
The information input to the information processing device 9 may be other than the command information.

[0037]

As described above, the present invention has the following effects.

(1) According to the first aspect of the present invention, it is possible to prevent an erroneous detection of an undesired object or an instructed object due to noise by checking whether the extracted tip point really belongs to the instructed object. Can be.

(2) If the straight line does not intersect the designated direction at one point in the space, the reference point is determined based on the sphere having the smallest radius among the spheres passing through the straight line.
The reference point can be determined correctly.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing one embodiment of the present invention.

FIG. 2 is an explanatory diagram illustrating an example of a verification method for a candidate for a tip point.

FIG. 3A is an explanatory view showing an example of a reference point search algorithm, and FIG. 3B is an enlarged view of a portion A in FIG. 3A.

FIG. 4 is a diagram illustrating an example of a method of determining a search range of a reference point.

[Explanation of symbols]

1, 2 TV camera 3 Information input device 4 Screen 5 Pointed target object 6 Hand 7 Tip point 8 Reference point 9 Information processing device 10 Operator 11 Binarized hand 6 12 Tip point candidate 14 Central pointed target object 15 ₁ １５15 ₄ Pointed target object 16 displayed at four corners 16 Tip point 17 Obtained straight line 18 ₁ -18 ₄ Obtained straight line 20 Length from index finger tip to first joint 21 Hand area in search path 23 22 Thickness of index finger 23 Circle (search path) 30 Estimated reference point 31 Radius of sphere centered on reference point 30 32 Sphere centered on reference point 30 40, 41, 42 Straight line 43, obtained by correction operation 43, 44 Nearest point between straight lines 40 and 42 45 Distance between closest points 43 and 44 46a, 46b Nearest point 47a, 47b between straight lines 41 and 42 Nearest point between straight lines 40 and 42 Xrange x-axis Search range Yrange y-axis search range

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-63-223913 (JP, A) JP-A-63-167923 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06F 3/033 310 G06T 1/00 G06T 7/00

Claims (2)

(57) [Claims] 1. A method of processing a plurality of imaging signals obtained from a plurality of imaging devices arranged so as to have different fields of view with respect to a pointing operation object, obtaining three-dimensional information of the pointing operation object, and obtaining the information. Input to the processing device, and determine the pointed target object existing on a straight line connecting a tip point obtained from a specific portion of the pointing motion object and a predetermined reference point, and this is sent to the information processing device. In the information input method, the point closest to the pointing target object, a point considered to be a part of the pointing motion object is set as the tip point candidate, and for the tip point candidate, on the imaging signal, A search is performed based on the tip point candidate, and it is determined from the search result and the feature amount of the pointing action object itself whether the tip point candidate corresponds to a part of the pointing action object and can be a tip point. Characterized by , How to enter information. 2. Processing a plurality of imaging signals obtained from a plurality of imaging devices arranged so as to have different fields of view with respect to the pointing operation object, obtaining three-dimensional information of the pointing operation object, and obtaining the three-dimensional information. Input to the processing device, and determine the pointed target object existing on a straight line connecting a tip point obtained from a specific portion of the pointing motion object and a predetermined reference point, and this is sent to the information processing device. In the inputting method of inputting information, the pointing operation object points to a plurality of known pointing target objects, thereby defining straight lines passing through the pointing target objects and the pointing operation objects, and converging these straight lines. Nearby, centering on a point in a three-dimensional space, determining the reference point based on the sphere having the smallest radius among spheres through which all the straight lines pass inside, input method.