JP6948363B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present invention relates to a technique for setting position / orientation information of a virtual object in a three-dimensional space in a mixed reality presentation system.

Consider a method of arranging a virtual three-dimensional model (hereinafter referred to as a three-dimensional virtual object) such as CAD data in a three-dimensional space. In the conventional method, the observer first projects a three-dimensional virtual object as a two-dimensional image on a two-dimensional screen such as a display, and then operates the position and orientation of the object using a keyboard or the like in a pseudo three-dimensional space. The arrangement of the three-dimensional virtual object was set. However, with this method, when projecting onto a two-dimensional image, information on the three-dimensional virtual object is lost, and it may be difficult to intuitively operate the three-dimensional virtual object and set the arrangement. In order to solve this problem, a method of manipulating and arranging a three-dimensional virtual object in a three-dimensional space using mixed reality (MR) technology (for example, Patent Document 1) has been attempted. ..

As an example of a method of manipulating a three-dimensional virtual object in a three-dimensional real space, there is a method of using a six-DOF position / orientation measuring device (for example, a stylus of a FASTRAK sensor manufactured by Polhemus). In this method, the observer takes out the position / posture change amount output from the 6-DOF position / posture measuring device, and sets the arrangement of the three-dimensional virtual object using the change amount as the moving rotation amount of the three-dimensional virtual object (patented). Reference 1). FIG. 4 is a schematic view showing how the observer performs the conventional operation method. Figure 4 shows a state in which the observer holds a 6-DOF position / posture measuring device in his hand and rotates and moves it as shown by the arrows in the figure, and this rotation and movement are reflected in the three-dimensional virtual object. ing.

As a means for presenting to the observer, an HMD (Head Mounted Display) or the like is used. With this HMD, the observer can easily grasp the positional relationship between the real space and the three-dimensional virtual object and set the arrangement of the three-dimensional virtual object.

Japanese Unexamined Patent Publication No. 2004-62758

Kenichi Hayashi, Hirokazu Kato, Shogo Nishida, "Front and Back Judgment of Real Objects and 3D Virtual Objects Using Borderline Base Stereo Matching", Journal of The Virtual Reality Society of Japan, Vol. 10, No. 3, pp. 371-380, 2005

Consider a case where an observer tries to set the position and orientation of a three-dimensional virtual object so that the arrangement of the real object and the three-dimensional virtual object imitating the shape of the real object are the same. At this time, since the intersection of the 6-DOF position / orientation measuring device and the real object is not a surface but a point, the attitude is not uniquely determined and it is difficult to accurately set the position / orientation of the 3D virtual object in a short time. ..

The present invention has been made in view of such a problem, and an object of the present invention is to accurately set the position and orientation of a three-dimensional virtual object in a short time.

As one means for achieving the above object, the information processing apparatus according to the present invention has the following configuration. That is, a setting means for setting a reference plane in the real space and a selection means for selecting one of the planes defining the area of the three-dimensional virtual object imitating the real object in the real space as the corresponding plane. , The associating means for associating the reference plane and the corresponding plane so as to have a predetermined positional relationship, the moving means for moving the position of the three-dimensional virtual object according to the movement of the reference plane, and the real object. It has a fixing means for fixing a relative position with respect to the three-dimensional virtual object.

According to the present invention, it is possible to accurately set the position and orientation of the three-dimensional virtual object in a short time.

The block diagram which shows the functional structure of an information processing apparatus. The schematic diagram which shows the functional structure of the information processing apparatus which concerns on Embodiment 1. The flowchart which shows the procedure of the process which concerns on Embodiment 1. The schematic diagram which shows the state of performing the conventional operation method. The schematic diagram which shows how the reference plane and the reference plane coordinate system are defined for a hand. A flowchart showing a detailed processing procedure for estimating the reference plane position / orientation. The schematic diagram which shows the sampling point of the contour of a hand. A flowchart showing a detailed processing procedure for determining whether a reference plane and a three-dimensional virtual object intersect. The figure which showed the example of the bounding box 901 of a three-dimensional virtual object 205. A flowchart showing a detailed processing procedure for intersecting the intersecting bounding box planes in parallel with the reference plane. The schematic diagram which shows the state that the bounding box 901 and the reference plane 501 intersect in parallel. A flowchart showing a detailed processing procedure for estimating the relative position and orientation of the reference plane and the virtual object plane. The figure which shows the state that the 3D virtual object 2050 is drawn according to the change of the reference plane coordinate system 502. The block diagram which shows the functional structure of the information processing apparatus which concerns on Embodiment 2. The schematic diagram which shows the functional structure of the information processing apparatus which concerns on Embodiment 2. The flowchart which shows the procedure of the process which concerns on Embodiment 2. A block diagram showing a computer hardware configuration applicable to an information processing device.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, based on its preferred embodiments. The configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration. The information processing device in the following embodiment sets the position / orientation information of a virtual object in a three-dimensional space in a mixed reality presentation system that fuses a virtual image by computer graphics or the like into a real space and presents it to an observer. It is an information processing device that can be used.

[Embodiment 1]
<Structure>
FIG. 1 is a block diagram showing a functional configuration of the information processing apparatus according to the present embodiment. Further, FIG. 2 is a schematic diagram of the information processing device according to the present embodiment, and FIG. 17 is a block diagram showing a computer hardware configuration applicable to the information processing device. In the present embodiment, as shown in FIG. 2, the image pickup unit 201, the display unit 202, and the image pickup unit position / orientation acquisition unit 211 are attached to the HMD (Head Mounted Display) 200, and each is fixed to the main body of the HMD 200. .. Two imaging units 201 and two display units 202 are provided on the main body of the HMD 200, the imaging unit 201R and the display unit 202R are for the right eye, and the imaging unit 201L and the display unit 202L are for the left eye. Hereinafter, R and L are omitted unless otherwise limited to those for the right eye and the left eye (that is, the imaging unit 201 and the display unit are 202). By attaching this HMD200 to the head, the observer can see an image (hereinafter referred to as MR image) in which the three-dimensional virtual object 205 input from the storage unit 104 is superimposed on the live-action image displayed on the display unit 202. Can be done.

The captured image acquisition unit 101 acquires the captured image captured by the imaging unit 201 as image data. The imaging unit position / orientation acquisition unit 211 estimates the viewpoint position / orientation of the observer from the result of measuring the position / orientation of the imaging unit 201. In this embodiment, a magnetic sensor is used as the image pickup unit position / orientation acquisition unit 211. The image pickup unit position / orientation acquisition unit 211 receives the magnetic field generated from the magnetic field generator 210 arranged in the real space, and sends the change of the received magnetic field to the sensor controller 209.

The sensor controller 209 obtains the position / orientation of the imaging unit position / orientation acquisition unit 211 in the reference coordinate system 206 with the magnetic field generator 210 as the origin. Here, the process of "obtaining the position and posture" is the following process. That is, first, the sensor controller 209 generates the _{position / orientation conversion matrix ΔM R} by using the change in the position / orientation measured by the image pickup unit position / orientation acquisition unit 211. After that, the sensor controller 209 integrates ΔM _{R into the matrix M R} indicating the position / orientation (three-dimensional arrangement information) of the image pickup unit position / orientation acquisition unit 211 in the previous frame, so that the image pickup unit position / orientation acquisition unit in the next frame the matrix M _R 'representing the position and orientation of the 211 obtains as the following equation (1).
(Equation 1)

（式3）

（式4）

（式2）は行列W_t（基準座標系206に対する姿勢を表す3×3回転行列を含む行列）、（式3）は行列W_r（基準座標系206に対する位置を表す3次元ベクトルを含む行列）を示す式である。行列Wtと行列Wrにおける各成分は周知の通り、撮像部位置姿勢取得部211による位置姿勢の計測値に基づいて決まるものである。 Incidentally, the matrix M _R ', the matrix W _t that indicates the position change component, the matrix W _r indicating the posture change component representing the position and orientation of the imaging unit position and orientation acquiring unit 211 are each a matrix of 4 × 4. The sensor controller 209 stores the initial position / orientation of the imaging unit position / orientation acquisition unit 211 in advance as the initial value of _{the matrix M R.} The position-orientation transformation matrix ΔM _R is a matrix calculated using the _{matrices W t} and W _r. The matrix W _t and the matrix W _r are obtained as outputs from the image pickup unit position / orientation acquisition unit 211.
(Equation 2)

(Equation 3)

(Equation 4)

(Equation 2) is the matrix W _t (a matrix containing a 3 × 3 rotation matrix representing the attitude with respect to the reference coordinate system 206), and (Equation 3) is a matrix W _r (a matrix containing a three-dimensional vector representing the position with respect to the reference coordinate system 206). ). As is well known, each component in the matrix Wt and the matrix Wr is determined based on the measured values of the position and orientation by the image pickup unit position and orientation acquisition unit 211.

Sensor controller 209, the position and orientation M _R 'of the image pickup unit position and orientation acquiring unit 211 with respect to a reference coordinate system 206, and outputs to the workstation 203. As shown in (Equation 5), the workstation 203 performs coordinate conversion using the known position / orientation acquisition unit 211 of the image pickup unit and the position / orientation matrix M _L of the image pickup unit 201, and the position / orientation with respect to the image pickup unit coordinate system 208. Find the matrix M _C that represents.
(Equation 5)

The sensor applicable to the image pickup unit position / orientation acquisition unit 211 is not limited to the magnetic sensor, and may be another type of sensor. For example, it may be an optical sensor, an ultrasonic sensor, or an inertial sensor. Further, if the position and orientation of the image pickup unit can be measured, such as the position and orientation measurement of the image pickup unit by image processing, the image pickup unit position and orientation acquisition unit 211 can be applied.

The reference plane position / orientation estimation unit 103 obtains the shape of the space including the hand 212 from one or more stereo images acquired by the captured image acquisition unit 101. Techniques for detecting the region of the hand and estimating the three-dimensional shape of the contour line are well known, and for example, the technique disclosed in Non-Patent Document 1 is used. According to this method, the reference plane position / orientation estimation unit 103 detects the object region by the difference from the key frame, and performs stereo matching of the boundary line of the object region. The reference plane position / orientation estimation unit 103 obtains the distance from the imaging unit 201 (camera) to the boundary line by triangulation for the matched boundary line. After that, the reference plane position / orientation estimation unit 103 estimates the position / orientation of the imaging unit 201 in the reference coordinate system 206, and obtains the position / orientation of the boundary line in the reference coordinate system 206. The reference plane position / orientation estimation unit 103 further obtains a reference plane 501 which is an approximate plane based on the position / orientation of the boundary line. FIG. 5 shows an example of the reference plane 501 obtained by the reference plane position / orientation estimation unit 103. The three-dimensional virtual object selection unit 102 determines whether or not an input for selecting the target three-dimensional virtual object 205 has been made from the input unit 109.

The storage unit 104 stores data (arrangement information, shape information, etc.) of the three-dimensional virtual object 205 obtained by imitating the shape of the mockup 204 measured by the workstation 203. The data of the three-dimensional virtual object 205 to be stored is composed of three-dimensional position information of vertices, a 4 × 4 matrix for coordinate transformation ( _{called a position-orientation matrix M CG} from the reference coordinate system 206), and the like. Further, the storage unit 104 stores the initial value of the position and orientation of the three-dimensional virtual object coordinate system 207 in the reference coordinate system 206. The storage unit 104 inputs the stored initial value to the intersection determination unit 105 as position information for arranging the three-dimensional virtual object 205 in the reference coordinate system 206. The storage unit 104 also stores the determination result of whether or not the target three-dimensional virtual object 205 is selected in the three-dimensional virtual object selection unit 102, and inputs the determination result to the three-dimensional virtual object arrangement fixing unit 110. The storage unit 104 can store data in any location, not limited to the external storage device 17070 shown in FIG. 17, as a hardware configuration.

The intersection surface determination unit 105 is based on the data of the reference surface 501 obtained from the reference surface position / orientation estimation unit 103 and the three-dimensional virtual object 205 stored in the storage unit 104, and the reference surface 501 and the three-dimensional virtual object 205. However, as an example of a predetermined positional relationship, it is determined whether or not they intersect. The relative position / orientation estimation unit 106 obtains the position / orientation of the reference plane coordinate system 502 of the reference plane 501 and the three-dimensional virtual object coordinate system 207 of the three-dimensional virtual object 205, respectively, and is relative to the two coordinate systems. Find the position and posture.

The three-dimensional virtual object position / orientation estimation unit 107 estimates the position / orientation of the three-dimensional virtual object 205 in the reference coordinate system 206. Position posture matrix M _SP reference plane coordinate system 502, using the relative position and orientation transformation matrix .DELTA.M _re, the position and orientation M _CG of the three-dimensional virtual object 205 in the reference coordinate system 206 'is determined by the following equation (6) ..
(Equation 6)

The image synthesizing unit 108 superimposes the three-dimensional virtual object 205 on the captured image acquired by the captured image acquisition unit 101. At this time, the image synthesizing unit 108 synthesizes the three-dimensional virtual object 205 from the viewpoint from the imaging unit 201 by using the position / orientation information of the imaging unit 201 and the position / orientation information of the three-dimensional virtual object 205. The input unit 109 notifies the three-dimensional virtual object selection unit 102 that there is an input to the workstation 203 from an input device such as a keyboard 17040, a mouse 17050, or a game pad.

The three-dimensional virtual object arrangement fixing unit 110 receives an image captured from the captured image acquisition unit 101 in response to an input from the input unit 109. The three-dimensional virtual object placement fixing unit 110 projects the captured image and the three-dimensional virtual object 205 on a two-dimensional plane, and changes the position and orientation of the three-dimensional virtual object 205 by fitting the mockup 204 captured in the captured image. Do it while letting it. The fitting is performed by a known method. For example, the features extracted from the virtual object may be matched with the features of the mockup included in the captured image. After fitting completion, the three-dimensional virtual object placement fixing unit 110 calculates the position and orientation M _CG of the three-dimensional virtual object 205 '. Three-dimensional virtual object placement fixing unit 110, the calculated position and orientation M _CG 'stored in the external storage device 17070, the transformation matrix M _CG of the three-dimensional virtual object coordinate system 207 in the reference coordinate system 206, the calculated position and orientation M _{Substitute CG} '.

<Processing procedure>
The processing procedure in this embodiment will be described with reference to the flowchart of FIG. In step S301, the captured image acquisition unit 101 acquires the image captured by the imaging unit 201. In step S302, the imaging unit position and orientation acquisition unit 211 obtains the matrix M _R 'representing the position and orientation of the imaging unit position and orientation acquiring unit 211 in the reference coordinate system 206. The workstation 203 then uses (Equation 5) to obtain the _{matrix M C} representing the position and orientation of the imaging unit coordinate system 208.

In step S303, the three-dimensional virtual object placement / fixing unit 110 uses the position / orientation matrix M _CG stored in the storage unit 104 in response to the input from the input unit 109 to refer to the three-dimensional virtual object 205 in the reference coordinate system 206. Place in. In step S304, the storage unit 104 checks whether or not an operation (input) for selecting the target three-dimensional virtual object 205 for setting the position / orientation has been performed from the three-dimensional virtual object selection unit 102. When the operation for selecting the target three-dimensional virtual object 205 is performed, the process proceeds to step S305. On the other hand, if the operation for selecting is not performed, the process proceeds to step S301.

In step S305, the captured image acquisition unit 101 inputs the captured image acquired from the imaging unit 201 to the reference plane position / orientation estimation unit 103, and the reference plane position / orientation estimation unit 103 estimates the position / orientation of the reference plane 501. FIG. 6 is a flowchart showing a detailed processing procedure of the position / orientation estimation step S305 of the reference plane 501. The processing procedure shown in FIG. 6 is based on the method described in Non-Patent Document 1. Further, FIG. 7 is a diagram simulating the sampling point 701 of the contour line extracted from the region of the hand 212. The sampling point 701 has information on the three-dimensional position in the image pickup unit coordinate system 208.

In step S601 of FIG. 6, the reference plane position / orientation estimation unit 103 estimates the position / orientation of the contour line in the three-dimensional space of the hand 212, and estimates the three-dimensional position of the sampling point 701.

求める近似平面上に存在する点は（式7）が成り立つ。（式7）に基づき、入力されたn個の点群がすべて近似平面上にあると仮定すると、（式8）が成り立つ。
（式8）

ここで、この式を、
（式9）

とおく。この（式9）のXの一般化逆行列を求めることで、近似平面の数式モデルのパラメータであるp(a, b, c)が求められる。ここで、（式9）のXの一般化逆行列をX⁺とすると
（式10）

となり、
（式11）

となる。 In step S602, the reference plane position / orientation estimation unit 103 calculates an approximate plane including the sampling point 701 using a mathematical model of an infinite plane by least squares approximation. If the equation of the plane is ax + by + cz = 1 and this is expressed by vector operation, it is expressed as (Equation 7).
(Equation 7)

(Equation 7) holds for the points existing on the approximate plane to be obtained. Based on (Equation 7), assuming that all the input n point clouds are on the approximate plane, (Equation 8) holds.
(Equation 8)

Here, this formula
(Equation 9)

far. By finding the generalized inverse matrix of X in (Equation 9), the parameters p (a, b, c) of the mathematical model of the approximate plane can be found. Here, if the generalized inverse matrix of X in (Equation 9) is X ⁺ , then (Equation 10)

Next,
(Equation 11)

Will be.

From this (Equation 11), the parameters p (a, b, c) of the mathematical model of the approximate plane can be obtained from the input point cloud. The sampling point 701 is set in the matrix of X in (Equation 11), and the parameters p (a, b, c) of the mathematical model of the approximate plane are obtained from the calculation result in (Equation 11). Here, by substituting the obtained parameters p (a, b, c) into (Equation 7), a mathematical expression of the approximate plane can be obtained. The method for obtaining the approximate plane is not limited to the method based on the least squares approximation, and other methods can also be applied.

In step S603, the reference plane position / orientation estimation unit 103 finds a rectangular parallelepiped circumscribing the three-dimensional shape of the contour line of the hand 212, finds the line of intersection with the approximate plane obtained in step S602, and determines the plane region of the hand 212. Then, this plane region is set as the reference plane 501. First, the reference plane position / orientation estimation unit 103 uses the maximum values of the X coordinate value, the Y coordinate value, and the Z coordinate value that can be taken by the sampling points 701 constituting the three-dimensional shape as a rectangular parallelepiped circumscribing the three-dimensional shape of the contour line. Calculate the formula for the 6 planes constructed using the minimum values. The reference plane position / orientation estimation unit 103 can obtain the line of intersection between the calculated 6 planes and the approximate plane obtained in step S602, and can record the line segment on the surface of the rectangular parallelepiped. Further, the reference plane position / orientation estimation unit 103 sets the intersections of the obtained plurality of line segments as the reference plane vertices 504A to 504D in the adjacent plane region, and constitutes the reference plane 501 from these reference plane vertices. At this time, the reference plane coordinate system 502 is defined with an arbitrary vertex of the reference plane 501 as the origin. In addition, the coordinate conversion matrix from the image pickup unit coordinate system 208 to the reference plane coordinate system 502 is set as M _CSP .

In the present embodiment, in order to set the plane region, the reference plane position / orientation estimation unit 103 obtains the circumscribing rectangular parallelepiped and obtains the plane region (reference plane 501) from the line of intersection between the rectangular parallelepiped and the approximate plane. rice field. However, the method is not limited to this method, and the reference plane position / orientation estimation unit 103 may calculate the plane region of the palm by using the three-dimensional shape of the contour line of the hand 212.

Returning to FIG. 3, in step S306, the relative position / orientation estimation unit 106 estimates the relative position / orientation between the reference plane 501 and the three-dimensional virtual object 205 estimated in step S305. In step S307, the intersection determination unit 105 determines whether the reference surface 501 estimated in step S305 intersects the three-dimensional virtual object 205. FIG. 8 is a flowchart showing a detailed processing procedure of step S307 in which the intersection surface determination unit 105 determines whether the reference surface 501 intersects the three-dimensional virtual object.

In step S801 of FIG. 8, the intersection determination unit 105 creates a bounding box for the three-dimensional virtual object 205. As a method of creating a bounding box of a three-dimensional virtual object 205, for example, there is a method of creating a rectangular parallelepiped that includes all three-dimensional points constituting the three-dimensional virtual object 205. FIG. 9 shows an example of the bounding box 901 of the three-dimensional virtual object 205.

In step S802, the intersection determination unit 105 determines whether the bounding box 901 and the reference surface 501 intersect. Specifically, each point of the bounding box 901 is compared with the position of a rectangular corner point constituting the reference surface 501 in the reference coordinate system 206. First, the intersection surface determination unit 105 obtains the maximum and minimum values of the X-axis, Y-axis, and Z-axis in the reference coordinate system 206 of the bounding box 901. The intersection surface determination unit 105 also obtains the maximum and minimum values of the X-axis, Y-axis, and Z-axis in the reference coordinate system 206 for the reference surface 501. The intersection surface determination unit 105 calculates whether the maximum value and the minimum value of the reference surface 501 are included in the range of the maximum value to the minimum value of the bounding box 901 for each axis. If it is included, the intersection surface determination unit 105 determines that it intersects. If it is determined that they intersect, the process is moved to step S803. If it is determined that they do not intersect, the processing flow is terminated. However, the algorithm for determining the intersection of the surface of the bounding box 901 and the reference surface 501 is not limited to this method, and the intersection may be determined by another method.

In step S803, the intersection determination unit 105 fits the surface of the bounding box 901 determined to intersect in S802 to the reference surface 501. FIG. 10 is a flowchart showing a detailed processing procedure of step S803 in which the intersection surface determination unit 105 fits (crosses in parallel) the surface of the bounding box 901 to the reference surface 501.

In step S1001 of FIG. 10, the intersection determination unit 105 calculates the distance between each surface of the bounding box 901 and the reference surface vertices 504A to 504D, and takes the average value of the distances from the reference surface vertices 504A to 504D to the surface. .. In step S1002, the intersection determination unit 105 obtains the average value of the distances from the reference surface vertices 504A to 504D for each surface of the bounding box 901. The surface with the shortest distance is designated as the corresponding surface 901A. In step S1003, the intersection determination unit 105 rotates and moves the three-dimensional virtual object 205 so as to intersect the corresponding surface 901A and the reference surface 501 in parallel. FIG. 11 shows how the bounding box 901 and the reference plane 501 intersect in parallel as a result of the movement.

Returning to FIG. 3, when the reference plane 501 and the three-dimensional virtual object 205 intersect as a result of the above processing in step S307, the processing proceeds to step S308. On the other hand, if they do not intersect, the process returns to step S301. In step S308, the three-dimensional virtual object position / orientation estimation unit 107 determines the relative position / orientation between the reference plane 501 estimated in step S306 and the three-dimensional virtual object 205 and the position of the reference plane 501 estimated in step S305. The position and orientation of the three-dimensional virtual object 205 is estimated from the orientation. FIG. 12 is a flowchart showing a detailed processing procedure of step S308 in which the three-dimensional virtual object position / orientation estimation unit 107 estimates the relative position / orientation between the reference surface 501 and the three-dimensional virtual object 205.

In step S1201 of FIG. 12, the three-dimensional virtual object position / orientation estimation unit 107 estimates the position / orientation of the reference plane coordinate system 502 in the reference coordinate system 206. Specifically, the three-dimensional virtual object position / orientation estimation unit 107 estimates the position / orientation matrix M _SP of the reference plane coordinate system 502 in the reference coordinate system 206 by the following (Equation 12). The matrix M _CSP is a coordinate conversion matrix from the image pickup unit coordinate system 208 to the reference plane coordinate system 502.
(Equation 12)

In step S1202, the three-dimensional virtual object position / orientation estimation unit 107 uses the coordinate conversion matrix M _{SP of the reference plane coordinate system 502 obtained in step S1201 and the coordinate conversion matrix M CG} of the three-dimensional virtual object coordinate system 207 in the reference coordinate system 206. As shown in (Equation 13), the relative position-attitude matrix M _re is calculated as follows.
(Equation 13)

FIG. 13 is a diagram in which the three-dimensional virtual object 205 is drawn using the estimated position and orientation. FIG. 13 shows how the three-dimensional virtual object 205 is drawn according to the change of the reference plane 501. From FIG. 13, it can be seen that the three-dimensional virtual object 205 is moving while the position and orientation of the three-dimensional virtual object 205 and the reference plane 501 are maintained.

Returning to FIG. 3, in step S309, when the three-dimensional virtual object placement fixing unit 110 refers to the input information from the input unit 109 and the determination of the placement of the three-dimensional virtual object 205 is input, the process proceeds to step S310. Move. Here, the arrangement is performed so that the contour line of the three-dimensional virtual object 205 coincides with the contour line of the mockup 204, and the position of the three-dimensional virtual object 205 is determined. On the other hand, if the arrangement decision is not input, the process proceeds to step S301. In step S310, the three-dimensional virtual object placement and fixing unit 110 stores the calculated position and orientation of the three-dimensional virtual object coordinate system 207 in the storage unit 104, and further substitutes the calculated position and orientation of the three-dimensional virtual object coordinate system 207 into the position and orientation of the three-dimensional virtual object coordinate system 207.

In step S311, when the end instruction of the main process is input or the condition for terminating the main process is satisfied, the main process is ended. On the other hand, if the end instruction of this process is not input and the condition for terminating this process is not satisfied, the process proceeds to step S301.

As described above, the information processing apparatus in the present embodiment sets the position and orientation of the three-dimensional virtual object so that the arrangement of the real object and the three-dimensional virtual object imitating the shape of the real object are the same. At this time, the information processing device operates the three-dimensional virtual object using another real object that can measure the position and orientation in the three-dimensional space. Since both the real object and the other real object are real objects, the posture of the three-dimensional virtual object can be uniquely determined by intersecting the real objects with each other on a surface. This makes it possible to set the position and orientation of the three-dimensional virtual object accurately and in a short time.

[Embodiment 2]
In the first embodiment, the position of the mockup 204 in the reference coordinate system 206 is not measured. Therefore, if the observer moves the mockup 204 after fixing the position and orientation of the three-dimensional virtual object 205 in step S310, the position and orientation of the three-dimensional virtual object 205 will deviate from the three-dimensional virtual object 205. Therefore, as the second embodiment, a mode in which the mockup coordinate system 1501 is defined in the mockup will be described. In the present embodiment, the three-dimensional virtual object 205 may rotate / move as the mockup 204 rotates / moves in the three-dimensional space.

FIG. 14 is a block diagram showing a functional configuration in the present embodiment. The same parts as in FIG. 1 are numbered the same, and the description thereof will be omitted. FIG. 15 is a schematic diagram of the configuration of the information processing device according to the present embodiment. The same parts as in FIG. 2 are numbered the same, and the description thereof will be omitted.

The mock-up position / attitude estimation unit 1401 estimates the position / orientation of the mock-up coordinate system 1501 in the reference coordinate system 206 from the position / orientation acquired by the mock-up position / attitude acquisition unit 1502. The relative position / orientation fixing unit 1402 calculates the relative position / orientation of the three-dimensional virtual object coordinate system 207 and the mockup coordinate system 1501 and stores the relative position / attitude in the storage unit 104.

FIG. 16 is a flowchart showing a processing procedure in the present embodiment. The same parts as in FIG. 3 are numbered the same, and the description thereof will be omitted. In step S1601, the three-dimensional virtual object placement and fixing unit 110 estimates the position and orientation of the three-dimensional virtual object coordinate system 207. In step S1602, the mockup position / orientation acquisition unit 1502 acquires the position / orientation in the reference coordinate system 206 of the mockup coordinate system 1501. In the present embodiment, for the sake of simplicity, it is assumed that the position / orientation of the mockup coordinate system 1501 is the same as the position / orientation obtained by the mockup position / attitude acquisition unit 1502. The mockup position / orientation acquisition unit 1502 calculates the relative position / orientation between the acquired mockup coordinate system 1501 and the three-dimensional virtual object coordinate system 207.

In step S1603, the relative position / orientation fixing unit 1402 substitutes the result of integrating the relative position / orientation obtained in S1602 with respect to the mockup coordinate system 1501 into the position / orientation of the three-dimensional virtual object coordinate system 207. As a result, the relative position and orientation of the mockup 204 and the three-dimensional virtual object 205 are fixed. In step S1604, the storage unit 104 refers to the input information from the input unit 109, and checks whether or not an operation (input) for selecting to fix the relative position / orientation is performed. If the operation of selecting to fix the relative position / orientation is performed, the process proceeds to step S1603. On the other hand, if the operation of selecting to fix the relative position / orientation has not been performed, the process proceeds to step S301.

As described above, the information processing apparatus in the present embodiment integrates the relative position / orientation calculated in step S1604 with respect to the mockup coordinate system 1501 when the mockup 204 rotates / moves in the three-dimensional space. As a result, the information processing device can obtain the position and orientation of the three-dimensional virtual object coordinate system 207.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Captured image acquisition unit, 102 3D virtual object selection unit, 104 Storage unit, 105 Crossing surface determination unit, 106 Relative position / orientation estimation unit, 107 3D virtual object position / orientation estimation unit, 108 Image composition unit, 109 Input unit, 110 3D virtual object placement fixed unit, 200 HMD, 201 imaging unit, 202 display unit, 203 workstation, 204 mockup, 205 3D virtual object, 206 reference coordinate system, 207 3D virtual object coordinate system, 208 imaging unit Coordinate system, 209 sensor controller, 210 magnetic field generator, 211 imaging unit position / orientation acquisition unit, 212 hands

Claims (13)

A setting means for setting a reference plane in the real space,
A selection means for selecting one surface as a corresponding surface from the surfaces defining the area of the three-dimensional virtual object that imitates the real object in the real space.
An associating means for associating the reference plane and the corresponding plane so as to have a predetermined positional relationship.
A moving means for moving the position of the three-dimensional virtual object according to the movement of the reference plane, and
A fixing means for fixing the relative position between the real object and the three-dimensional virtual object,
An information processing device characterized by having. The information processing device according to claim 1, wherein the setting means sets a surface that approximates the shape of a real operation object to be operated in the real space as the reference surface. The setting means, the information processing apparatus according to claim 2, characterized in that for calculating the plane including the sampling points of the real manipulation object outline on the real space as the reference plane. The method according to any one of claims 1 to 3, wherein the selection means selects, among the surfaces of the three-dimensional virtual object, the surface having the shortest distance from the reference surface as the corresponding surface. The information processing device described in. The selection means is characterized in that a bounding box of the three-dimensional virtual object is created, and among the surfaces of the bounding box, the surface having the shortest distance from each vertex of the reference surface is selected as the corresponding surface. The information processing apparatus according to any one of claims 1 to 4. The information processing apparatus according to any one of claims 1 to 5, wherein the associating means associates the reference plane and the corresponding plane so as to intersect in parallel. The information processing apparatus according to any one of claims 1 to 6, wherein the associating means associates the three-dimensional virtual object by rotating and moving the three-dimensional virtual object. The information processing apparatus according to any one of claims 1 to 7, wherein the moving means moves the position of the three-dimensional virtual object so as to coincide with the contour line of the real object. The information processing apparatus according to claim 2 or 3 , further comprising a display means for displaying an image based on the three-dimensional virtual object based on the position and orientation of the viewpoint of the observer. The display means includes an imaging unit that captures an image of the real space, and displays the image of the three-dimensional virtual object superimposed on the image of the real space captured by the imaging means. Item 9. The information processing apparatus according to Item 9. The setting means, the information processing apparatus according to claim 10, characterized in that said real manipulation object on the physical space from a plurality of images captured by the imaging means, create and set the reference plane .. The setting process of setting the reference plane in the real space and
A selection step of selecting one surface as a corresponding surface from the surfaces defining the area of a three-dimensional virtual object that imitates a real object in the real space.
An association step of associating the reference surface and the corresponding surface so as to have a predetermined positional relationship,
A moving step of moving the position of the three-dimensional virtual object according to the movement of the reference plane, and
A fixing step of fixing the relative position between the real object and the three-dimensional virtual object,
An information processing method characterized by having. A program for operating a computer as each means of the information processing apparatus according to any one of claims 1 to 11.

Images