JP5874698B2 - Information processing apparatus, control method thereof, and program - Google Patents

Description

The present invention relates to an information processing apparatus capable of causing a user to recognize contact between a three-dimensional model that moves following an object held by the user and a three-dimensional model different from the three-dimensional model, a control method thereof, And the program.

  Conventionally, an image of a virtual world (hereinafter referred to as a virtual space) generated by a computer is superimposed on an image of a real world (hereinafter referred to as a real space) taken by a head mounted display (HMD) or a smartphone, and then the HMD or smartphone. There are technologies to display on the screen. For example, there are mixed reality (hereinafter referred to as MR) technology and augmented reality technology.

  In particular, the MR technology superimposes and displays the real space image and the virtual space image on the HMD worn by the user, so that it appears as if the virtual space information is displayed in front of the eyes, and there is a sense of presence. Very expensive. Therefore, various utilization methods using MR technology are considered.

  For example, in Patent Document 1 described below, in a mixed reality presentation device that nurtures a virtual pet, the internal state of a three-dimensional model indicating a virtual pet is changed by a predetermined operation by a predetermined part of a user, thereby A technique for facilitating the recognition of presence is disclosed.

JP 2002-112286 A

  By the way, there are cases where a user moves a three-dimensional model arranged in a virtual space using this MR technique. For example, when moving furniture in a model room expressed by MR technology.

  More specifically, the image of the real space and the image of the virtual space in which the model room is represented by a three-dimensional model are superimposed, and the superimposed image is displayed through the HMD, thereby reproducing the actually constructed room. To do. In the displayed model room, furniture such as a chair and a desk expressed in a three-dimensional model is arranged, and the user can freely carry this furniture through a device such as a mouse. By doing this, you can experience the model room while laying out your favorite furniture in your favorite position.

  However, when furniture is moved in such a model room, the model room and the furniture are both represented by a three-dimensional model, and therefore, the furniture can be moved through the wall. Originally, control should be performed so that the three-dimensional models do not overlap as in the real space. However, conventionally, since a 3D model is displayed at a specified position in a virtual space, there is a problem that even if 3D models overlap each other, they are displayed as they are at a specified position. is there. That is, it is required to realize a realistic operation similar to that in the real space with the MR technology.

This onset Ming, to provide a three-dimensional model which moves following the object Yoo over THE holds, a possible mechanism allowing the user to recognize contact with a different 3-dimensional model with the three-dimensional model For the purpose .

In order to achieve the above object, the information processing apparatus of the present invention provides a mixed reality image obtained by superimposing a virtual space image, which is an image obtained by capturing a virtual space, on a real space image, which is an image obtained by capturing the real space, An information processing apparatus for presenting a real space image, and a real space image input receiving means for receiving an input of the real space image, and a position / posture input reception for receiving an input of a position and posture of a user presenting the mixed reality image And a setting unit configured to set the three-dimensional model arranged in the virtual space to follow the movement of the object held by the user in the real space in response to a first instruction from the user; 3D model which has been set in a 3-dimensional model moving means for moving so as to follow the movement of the object, a three-dimensional model that has been moved in the three-dimensional model moving means A virtual space image generating means for generating the virtual space image by capturing an image of the virtual space based on at least one of the position and orientation received by the position / orientation input receiving means, and the real space image input receiving A virtual reality image generated by the virtual space image generation means is superimposed on the real space image received by the means to generate the mixed reality image, and the setting means includes the setting means When the three-dimensional model moved following the movement of the object comes into contact with a three-dimensional model different from the three-dimensional model, the setting for following the movement of the object set by the setting unit is canceled, the three-dimensional model moving means, in response to the setting to follow the motion of the object by the setting means is canceled, the three-dimensional model is moved, the user to hold Not follow the movement of the object, characterized in that stop at a position come in contact with different three-dimensional model with the three-dimensional model.

According to the present invention, it is possible to recognize a user contact with a different 3-dimensional models and three-dimensional model which moves following the object Yoo over THE holds, with the 3-dimensional model.

It is a block diagram which shows an example of MR system 100 at the time of using a marker. It is a block diagram which shows an example of MR system 100 at the time of using a magnetic tracking sensor. It is a block diagram which shows an example of MR system 100 at the time of using an optical tracking sensor. It is a schematic diagram which shows an example of the mouse | mouth 401 used with MR system 100 using an optical tracking sensor. 2 is a configuration diagram illustrating an example of a hardware configuration of an MR system 100. FIG. 2 is a configuration diagram illustrating an example of a module configuration of an information processing apparatus 101. FIG. It is a flowchart which shows an example of the flow of a series of processes in embodiment of this invention. It is a flowchart which shows an example of the virtual space production | generation process in embodiment of this invention. 6 is a schematic diagram illustrating an example of a case where a cursor object 901 is superimposed on a mouse 401. FIG. FIG. 10 is a schematic diagram illustrating an example when a virtual object 1002 is lifted by a cursor object 901. FIG. 6 is a schematic diagram illustrating an example of a case where a virtual object 1002 lifted by a cursor object 901 and a virtual object 1003 come into contact with each other, and a case where the virtual object 1002 is drawn.

  Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

  1 to 3 are diagrams showing an example of the system configuration of the MR system 100. FIG. In an MR system 100 shown in FIGS. 1 to 3, an information processing apparatus 101 and an HMD 102 are connected to each other so that data communication can be performed. The connection between the information processing apparatus 101 and the HMD 102 may be a wired connection or a wireless connection. Note that the configurations of various terminals connected on the system shown in FIGS. 1 to 3 are merely examples, and there are various configuration examples depending on applications and purposes.

  The information processing apparatus 101 is an apparatus such as a personal computer or a server. The information processing apparatus 101 superimposes a real space image (hereinafter referred to as a real space image) captured (captured) by the HMD 102 and a virtual space image (hereinafter referred to as a virtual space image) generated by the information processing apparatus 101. An image (hereinafter, mixed reality image) is generated and transmitted to the HMD 102.

  The HMD 102 is a so-called head mounted display. The HMD 102 is a display device worn on the user's head, and includes a right-eye and left-eye video camera, and a right-eye and left-eye display. The HMD 102 transmits a real space image captured by the video camera of the HMD 102 to the information processing apparatus 101. Then, the mixed reality image transmitted from the information processing apparatus 101 is received and displayed on the display. Further, since the right-eye and left-eye video cameras and the display are provided, a stereoscopic effect can be obtained by parallax. The physical space image captured by the HMD 102 and the mixed reality image to be displayed are preferably in the moving image format, but may be images captured at a predetermined interval.

  First, in the MR system 100 of FIG. 1, a uniquely identifiable marker 103 is installed at a predetermined location, and the information processing apparatus 101 identifies the marker 103 included in the real space image transmitted from the HMD 102. This is a system that identifies the position and orientation of the HMD 102. More specifically, a predetermined position in the virtual space and a predetermined marker 103 are linked, and when the marker 103 is detected from the real space image, the HMD 102 is calculated from the size and inclination of the marker 103. Is located in the virtual space and in what posture. In the MR system 100, it is preferable to arrange the marker 103 so that any marker 103 can be detected regardless of which direction the HMD 102 faces.

  Next, the MR system 100 of FIG. 2 is a system that identifies the position and orientation of the HMD 102 using a transmitter 201 that generates a magnetic field. The transmitter 201 generates a magnetic field, and the generated magnetic field is received by a receiver 202 provided in the HMD 102. When information on the received magnetic field is transmitted to the information processing apparatus 101, the information processing apparatus 101 specifies the position and orientation of the HMD 102 from the strength of the magnetic field.

  The MR system 100 in FIG. 3 is a system that identifies the position and orientation of the HMD 102 using the infrared camera 301 (sensor). The HMD 102 is provided with an optical marker 302 called an optical marker 302. When an image is captured using the infrared camera 301 and the captured result is transmitted to the information processing apparatus 101, the optical marker 302 is detected and the position and orientation of the HMD 102 are specified. To do. It is desirable to install a plurality of infrared cameras 301 so that the optical marker 302 can be photographed.

  Thus, in order to specify the position and orientation of the HMD 102, it is necessary to use a marker or a sensor. Any form may be used as long as the position and posture of the HMD 102 can be specified. In the present embodiment, the system configuration in FIG. 3 will be described as an example.

  FIG. 4 is an example of a mouse 401 used in the present embodiment. The mouse 401 is a pointing device and is connected to the information processing apparatus 101 so as to be able to perform data communication. The connection between the mouse 401 and the information processing apparatus 101 may be a wired connection or a wireless connection. The mouse 401 includes a first button 402 and a second button 403. Further, the mouse 401 is provided with an optical marker 302 similar to that of the HMD 102. With this optical marker 302, not only the HMD 102 but also the position and posture of the mouse 401 can be specified.

  FIG. 5 is a diagram illustrating a hardware configuration of the information processing apparatus 101 and the HMD 102 in the embodiment of the present invention. Note that the hardware configurations of the information processing apparatus 101 and the HMD 102 in FIG. 5 are merely examples, and there are various configuration examples depending on applications and purposes.

  First, the information processing apparatus 101 includes a CPU 501, a ROM 502, a RAM 503, a system bus 504, an input controller 505, a video controller 506, a memory controller 507, a communication I / F controller 508, an input device 509, a display 510, an external memory 511, and the like. .

  The CPU 501 comprehensively controls each device and controller connected to the system bus 504.

  The ROM 502 or the external memory 511 stores a BIOS (Basic Input / Output System) that is a control program of the CPU 501, an operating system, and various programs described below that are necessary for realizing the functions executed by various devices. Yes. The RAM 503 functions as a main memory, work area, and the like for the CPU 501.

  The CPU 501 implements various operations by loading a program or the like necessary for execution of processing into the RAM 503 and executing the program.

  An input controller (input C) 505 controls input from a pointing device such as a keyboard or mouse 401.

  A video controller (VC) 506 controls display on a display device such as the display 510. The display device may be a liquid crystal display or a CRT.

  The memory controller (MC) 507 is an adapter to a hard disk (HD), flexible disk (FD) or PCMCIA card slot for storing boot programs, browser software, various applications, font data, user files, editing files, various data, and the like. Controls access to an external memory 511 such as a card-type memory connected via the.

  A communication I / F controller (communication I / FC) 508 connects and communicates with an external device via a network, and executes communication control processing in the network. For example, Internet communication using TCP / IP is possible. The communication I / F controller 508 also controls communication with the receiver 202 that has received the magnetic field from the transmitter 201 and communication with the infrared camera 301.

  The CPU 501 enables display on the display 510 by, for example, executing outline font rasterization processing on the display information area in the RAM 503. Further, the CPU 501 enables a user instruction with a mouse cursor (not shown) on the display 510.

  Various programs used by the information processing apparatus 101 of the present invention to execute various processes described later are recorded in the external memory 511 and executed by the CPU 501 by being loaded into the RAM 503 as necessary. is there. Furthermore, definition files and various information tables used by the program according to the present invention are stored in the external memory 511.

  Next, the HMD 102 includes a right eye video camera 521, a left eye video camera 522, a right eye display 523, a left eye display 524, a controller 525, and the like.

  The right-eye video camera 521 and the left-eye video camera 522 are video cameras that capture the real world. The right eye video camera 521 captures an image to be displayed on the right eye display 523, and the left eye video camera 522 captures an image to be displayed on the left eye display 524. The captured image (real space image) is transmitted by the controller 525 to the information processing apparatus 101 and received by the information processing apparatus 101 through the communication I / F controller 508.

  When the mixed reality image is transmitted from the information processing apparatus 101 through the communication I / F controller 508, the controller 525 receives the mixed reality image and displays the received mixed reality image on the right-eye display 523 and the left-eye display 524. At this time, the mixed reality image generated based on the real space image photographed by the right eye video camera 521 is displayed on the right eye display 523, and the complex reality image generated based on the real space image photographed by the left eye video camera 522 is displayed. The real image is displayed on the left eye display 524.

  In the case of the system configuration as shown in FIG. 2, the receiver 202 is installed in the HMD 102, and the magnetic field information received by the receiver 202 is transmitted from the receiver 202 to the information processing apparatus 101.

  FIG. 6 is a functional configuration diagram illustrating a module configuration of the information processing apparatus 101. Note that the module configuration of the information processing apparatus 101 in FIG. 6 is merely an example, and there are various configuration examples depending on applications and purposes.

  The information processing apparatus 101 includes an operating system 600, an MR platform 610, and an MR application 620.

  The operating system 600 includes a communication module 601 and an input device control module 602. The communication module 601 is a module for communicating with the HMD 102, the receiver 202, and the infrared camera 301. The communication module 601 communicates information with various apparatuses through the communication I / F controller 508. The input device control module 602 is a module that receives a signal from the input device 509 such as the mouse 401 and performs various controls in the information processing apparatus 101.

  The MR platform 610 includes a calibration module 611, an image processing module 612, a marker detection module 613, a magnetic field analysis module 614, an infrared information analysis module 615, a position / orientation identification module 616, and a color detection module 617.

  The calibration module 611 is a module for aligning the real space and the virtual space. The real space and the virtual space are aligned based on the real space image photographed by the HMD 102 and information from various sensors.

  The image processing module 612 is a module that executes image processing to extract the marker 103 and the like from the real space image photographed by the HMD 102. The content of the image processing is not particularly limited. Any image processing that makes it easy to extract necessary information from the real space image may be used.

  The marker detection module 613 is a module for detecting the marker 103 from the real space image photographed by the HMD 102 and image-processed by the image processing module 612. The feature points of the marker 103 are extracted from the real space image, and the size and inclination of the marker 103 are further detected. Information of the detected marker 103 is transferred to a position / orientation specifying module 616 described later.

  The magnetic field analysis module 614 is a module for analyzing the magnetic field generated by the transmitter 201 transmitted by the receiver 202. The strength and change of the magnetic field are analyzed and passed to the position / orientation specifying module 616 described later.

  The infrared information analysis module 615 is a module for analyzing an infrared electrical signal (hereinafter referred to as infrared information) transmitted from the infrared camera 301. The optical marker 302 is detected from the infrared information, and the detected information is passed to a position / orientation specifying module 616 described later.

  The position / orientation specifying module 616 is a module that specifies the position and orientation of the HMD 102 or the mouse 401 based on various information passed from the marker detection module 613, the magnetic field analysis module 614, and the infrared information analysis module 615. Since the real space and the virtual space are aligned by the calibration module 611, the position in the virtual space can be specified from the position in the real space.

  The color detection module 617 is a module for detecting a predetermined color from the real space image transmitted from the HMD 102. In particular, the mask setting module 622, which will be described later, detects skin-colored pixels so that the user's hand is not hidden by the three-dimensional model. A color value close to the skin color may be set by RGB (Red-Green-Blue color model), and the same or similar color may be determined as the skin color.

  The MR application 620 includes a virtual camera control module 621, a mask setting module 622, a virtual object management module 623, and a superimposed image generation module 624.

  The virtual camera control module 621 is a module that controls the position and orientation of the camera in the virtual space based on the position and orientation in the virtual space identified by the position / orientation identification module 616. The camera in the virtual space is a camera for photographing the virtual space. That is, it is a user's viewpoint in the virtual space. A mixed reality image is generated by superimposing the virtual space image captured by the camera and the real space image transmitted from the HMD 102.

  The mask setting module 622 is a module that sets a mask for pixels that do not superimpose a virtual space image among pixels constituting the real space image when the virtual space image and the real space image are superimposed. The virtual space image is not superimposed on the pixel for which the mask is set. In particular, a mask is set for the pixel of the color detected by the color detection module 617. If the skin color is detected, a mask is set in the skin color pixel portion so that the user's hand can be seen at any time.

  The virtual object management module 623 is a module for managing a three-dimensional model (hereinafter referred to as a virtual object) arranged in the virtual space. It manages the placement positions of virtual objects in the virtual space, displays, moves, and hides virtual objects, links virtual objects, determines contact between virtual objects, and the like.

  The superimposed image generation module 624 superimposes the real space image transmitted from the HMD 102 and the virtual space image captured by the virtual space camera controlled by the virtual camera control module 621 to generate a mixed reality image. It is a module. At this time, the virtual space image is not superimposed on the pixels of the real space image for which the mask is set by the mask setting module 622. In addition, the superimposed image generation module 624 generates a mixed reality image corresponding to the real space image captured by the right eye video camera 521 and the left eye video camera 522, respectively. Note that the processes executed by the various modules included in the information processing apparatus 101 on the real space image captured by the right eye video camera 521 and the real space image captured by the left eye video camera 522 are the same.

  Next, a series of processing performed by the information processing apparatus 101 in the embodiment of the present invention will be described with reference to the flowchart shown in FIG.

  In step S701, the CPU 501 of the information processing apparatus 101 activates the MR platform 610 and the MR application 620 stored in the external memory 511 in response to an instruction from the user, and accepts transmission of information from the HMD 102 and the infrared camera 301. .

  In step S <b> 702, the controller 525 of the HMD 102 captures a real space with the right eye video camera 521 and the left eye video camera 522. In step S <b> 703, the controller 525 of the HMD 102 transmits the captured real world image (real space image) to the information processing apparatus 101. In the present embodiment, the images taken by the right eye video camera 521 and the left eye video camera 522 may be still images or moving images. In order to experience a more realistic mixed reality, animation is better.

  In step S704, the CPU 501 of the information processing apparatus 101 receives the real space image transmitted from the HMD 102 using the communication module 601, and stores it in the RAM 503 (real space image input receiving unit).

  On the other hand, in step S705, the infrared camera 301 images the real space. At this time, since the infrared camera 301 captures infrared light, the above-described optical marker 302 attached to the HMD 102 and the mouse 401 is captured. In step S <b> 706, the infrared camera 301 transmits the captured infrared electrical signal (infrared information) to the information processing apparatus 101.

  In step S <b> 707, the CPU 501 of the information processing apparatus 101 receives the infrared information transmitted from the infrared camera 301 using the communication module 601 and stores it in the RAM 503 (position / attitude input receiving unit). Since the infrared information is information indicating the position and orientation of the optical marker 302 attached to the HMD 102 and the mouse 401, the information processing apparatus 101 accepts the user wearing the HMD 102 and the input of the position and orientation of the mouse 401. It is synonymous with.

  In step S <b> 708, the CPU 501 of the information processing apparatus 101 receives an operation signal from the mouse 401 using the input device control module 602. More specifically, when a button included in the mouse 401 is pressed (clicked), an operation signal related to the pressed button is transmitted from the mouse 401 to the information processing apparatus 101, and the information processing apparatus 101 performs input device control. This is received through module 602. In the present embodiment, the first button 402 and the second button 403 included in the mouse 401 are used, but other buttons may be used.

  In step S <b> 709, the CPU 501 of the information processing apparatus 101 determines whether an end instruction has been given to the MR platform 610 or the MR application 620 from the user. When the menu for ending the MR platform 610 or the MR application 620 is selected by the user, it may be determined that there has been an ending instruction, or other than that. If it is determined that there is an end instruction, the process proceeds to step S719. If it is not determined that there is an end instruction, the process proceeds to step S710.

  In step S710, the CPU 501 of the information processing apparatus 101 uses the infrared information analysis module 615 to detect the position and orientation of the optical marker 302 provided in the HMD 102 and the mouse 401 from the infrared information received in step S707. In addition, about the method to detect from infrared information, since the existing technique is used, detailed description is abbreviate | omitted.

  In step S711, the CPU 501 of the information processing apparatus 101 uses the position / orientation specifying module 616 to determine the positions of the HMD 102 and the mouse 401 in the virtual space from the positions and orientations of the HMD 102 and the optical marker 302 of the mouse 401 detected in step S710. And identify posture. That is, the position and posture in the real space of the user wearing the HMD 102 and the mouse 401 operated by the user are specified. The calibration module 611 has adjusted in advance so that the predetermined position in the real space and the predetermined position in the virtual space coincide. Therefore, if the position and orientation in the real space can be specified, the position and orientation in the virtual space can also be specified.

  In step S 712, the CPU 501 of the information processing apparatus 101 receives the actual operation received in step S 704 based on the mouse operation signal received in step S 708 and the positions and postures of the HMD 102 and mouse 401 specified in step S 711 in the virtual space. A process of generating a virtual space to be superimposed on the space image is executed. Details of the virtual space generation processing will be described later with reference to FIG.

  In step S713, the CPU 501 of the information processing apparatus 101 uses the virtual camera control module 621 to control the position and orientation of the virtual camera in the virtual space, and shoots the virtual space (virtual space image generation unit). Then, the virtual space image generated by photographing is stored in the RAM 503.

  More specifically, the position and orientation of the HMD 102 in the virtual space is specified from the position and orientation of the HMD 102 specified in step S711, and a virtual camera is installed at the specified position and direction. . The virtual camera is a virtual photographing unit that is installed in the virtual space and photographs the virtual space. Since the virtual space is a virtual three-dimensional space formed from virtual objects, in order to display this on the HMD 102, it must be a two-dimensional image. Therefore, a two-dimensional image is acquired by photographing a three-dimensional space from a virtual camera. Note that the shooting here is to capture a virtual space. When capturing, only virtual objects that exist in the virtual space are captured. Therefore, the background is transparent.

  Further, even when a mask object is arranged in front of the virtual object when viewed from the virtual camera, the virtual object in the portion where the mask object overlaps is transmitted. The mask object is one of virtual objects that are installed in the virtual space by the user. By arranging the mask object, the problem that the virtual object is always displayed in the foreground can be prevented. For example, there is a case where a virtual object is desired to be displayed beyond an object placed in the real space. At this time, if the mask object is not arranged, the virtual space image is superimposed on the real space image in step S715 described later, so that the virtual object is displayed in front of the object. Therefore, a mask object is set at a position in the virtual space corresponding to the position of the object. In this way, a portion of the virtual object that is behind the mask object as viewed from the virtual camera passes through the portion overlapping the mask object. Thereby, even if the virtual space image is superimposed on the real space image, it appears that the virtual object is displayed behind the object.

  In step S <b> 714, the CPU 501 of the information processing apparatus 101 acquires the virtual space image captured in step S <b> 713 from the RAM 503 using the virtual camera control module 621.

  In step S715, the CPU 501 of the information processing apparatus 101 uses the superimposed image generation module 624 to superimpose the virtual space image acquired in step S714 on the real space image received in step S704, thereby generating a mixed reality image. (Mixed reality image generation means). At this time, the CPU 501 of the information processing apparatus 101 uses the color detection module 617 to detect pixels including a predetermined color from the real space image, and further uses the mask setting module 622 to detect a virtual space image on the detected pixels. Is controlled so as not to be superimposed. For example, the skin color of the user's hand is detected, and the virtual space image is not superimposed on the detected part. In this way, the user's hand appears to be displayed in front of the virtual object.

  In step S716, the CPU 501 of the information processing apparatus 101 uses the communication module 601 to transmit the mixed reality image generated in step S715 to the HMD 102 (mixed reality image transmission unit). Then, the process returns to step S704, and the processes in steps S702 to S718 are repeated until an end instruction is issued in step S709.

  In step S717, the controller 525 of the HMD 102 receives the mixed reality image transmitted in step S716. In step S718, the controller 525 of the HMD 102 displays the mixed reality image received in step S717 on the right eye display 523 and the left eye display 524, and presents the mixed reality image to the user (display unit). Although explanation is omitted in the above-described processing, two mixed reality images are generated, one displayed on the right eye display 523 and the other displayed on the left eye display 524. What is displayed on the right eye display 523 displays a mixed reality image generated using the real space image acquired from the right eye video camera 521. On the other hand, what is displayed on the left-eye display 524 displays a mixed reality image generated using the real space image acquired from the left-eye video camera 522.

  Thus, by repeatedly executing Steps S702 to S718, the user can experience mixed reality in real time. If it is determined in step S709 that an instruction for termination has been given, the process proceeds to step S719. In step S719, the CPU 501 of the information processing apparatus 101 terminates the MR platform 610 and the MR application 620.

  Next, virtual space generation processing performed by the information processing apparatus 101 according to the embodiment of this invention will be described using the flowchart shown in FIG.

  The virtual space generation process in the present invention is a process for generating a virtual space composed of virtual objects, and is a process for controlling the position and movement of the virtual object in the virtual space in accordance with the contents instructed by the user through the mouse 401. is there.

  An overview of the processing in FIG. 8 will be described with reference to FIGS. 10 and 11. First, 1000 in FIG. 10 will be described. Reference numeral 1000 denotes an initial state in which virtual objects are arranged in the virtual space. A user 1001 is a user wearing the HMD 102 and browsing a mixed reality image. A virtual object 1002 and a virtual object 1003 are virtual objects arranged in a virtual space. The cursor object 901 is a cursor object placed in step S802 described later. A cursor object is a three-dimensional model for detecting contact with a virtual object. The cursor object is displayed so as to be superimposed on the mouse 401 as indicated by a cursor object 901 in FIG. In a process to be described later, a virtual object in contact with the cursor object 901 can be operated. The user 1001 is viewing the cursor object 901, the virtual object 1002, and the virtual object 1003 superimposed on the real space image through the right eye display 523 and the left eye display 524 of the HMD 102.

  First, the user 1001 moves to bring the mouse 401 close to the virtual object, and brings the cursor object 901 into contact with the virtual object. Then, the user 1001 presses the first button 402 of the mouse 401 while in contact. Then, the CPU 501 of the information processing apparatus 101 detects that the button has been pressed, and performs control so that the touching virtual object follows the cursor object 901. 1010 in FIG. 10 is a diagram in which the virtual object 1002 is lifted by pressing the first button 402 in a state where the cursor object 901 is in contact with the virtual object 1002. As described above, while the CPU 501 of the information processing apparatus 101 continues to detect pressing of the first button 402, the virtual object 1002 can be moved so as to follow the cursor object 901.

  Next, when the user 1001 moves while pressing the first button 402 with the mouse 401 and passes through the virtual object 1003, in this embodiment, as shown by 1100 in FIG. The tracking is stopped in a state where the virtual object 1002 that is following is in contact with the virtual object 1003. This is because even if it is a virtual object, a realistic virtual space is presented by performing an operation closer to the real space.

  In the conventional mixed reality technology, since virtual objects are data only, virtual objects can overlap each other. This is because the CPU 501 of the information processing apparatus 101 has arranged the virtual object at the position of the coordinate value designated for each virtual object. That is, the virtual object is displayed at the designated position without considering the interference between the virtual objects. However, when designing a car or the like using mixed reality technology, it is not known whether or not the part can actually be incorporated unless the interference between the parts that are virtual objects is checked. In another case, when a model room is presented with mixed reality technology, if you try to carry furniture, which is a virtual object, into the room, you must check whether the furniture can pass through the door of the room. End up with no simulation.

  Therefore, when the virtual object 1002 moved by the user 1001 comes into contact with another virtual object 1003, the moving virtual object 1002 comes into contact with the other virtual object 1003 by stopping the virtual object 1002. This is notified to the user.

Further, as indicated by 1110 in FIG. 11, when the virtual objects are in contact with each other as in 1100 and the first button 402 of the mouse 401 is pressed, the second button 403 of the mouse 401 is further pressed. Then, the virtual object 1002 resumes following the cursor object 901. When the tracking of the virtual object 1002 is stopped, the user 1001 has a trouble of taking the virtual object 1002 again. Therefore, in order to easily move the virtual object 1002, when the second button 403 is pressed while the first button 402 is pressed, the tracking of the cursor object 901 is resumed.
Hereinafter, this series of processes will be described.

  First, in step S801, the CPU 501 of the information processing apparatus 101 uses a virtual object management module to place a virtual object at a predetermined position in the virtual space. Which virtual object is to be placed at which position in the virtual space is designated in advance by the user. Therefore, a predetermined virtual object is arranged at a position according to the designation. There may be a plurality of virtual objects.

  In step S802, the CPU 501 of the information processing apparatus 101 arranges the cursor object 901 at the position of the virtual space corresponding to the position of the mouse 401 specified in step S711 described above.

  In step S803, the CPU 501 of the information processing apparatus 101 uses the input device control module 602 to press the first button 402 (first instruction) or the second button 403 (second instruction) provided on the mouse 401. It is determined whether or not it has been done. More specifically, the determination is made based on whether a signal indicating that the first button 402 or the second button 403 is pressed is detected. If it is determined that the first button 402 or the second button 403 is pressed, the process proceeds to step S403. If it is determined that neither the first button 402 nor the second button 403 is pressed, the process proceeds to step S819.

  In step S <b> 804, the CPU 501 of the information processing apparatus 101 determines whether the cursor object 901 has a child element using the virtual object management module 623. The child element is a virtual object that follows the cursor object 901 like the virtual object 1002 described above. That is, it is determined whether there is a virtual object moved by the cursor object 901 in accordance with an instruction from the user. If it is determined that there is a child element, the process proceeds to step S805. If it is determined that there is no child element, the process proceeds to step S807.

  In step S805, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to determine whether or not the cursor object 901 is in contact with a virtual object (hereinafter referred to as ObjX) that follows the cursor object 901. That is, it is determined whether or not the first button 402 or the second button 403 is pressed by the user to move the ObjX. If it is determined that the cursor object 901 is in contact with ObjX, the process proceeds to step S806. If it is determined that the cursor object 901 is not in contact with ObjX, the process proceeds to step S814.

  In step S <b> 806, the CPU 501 of the information processing apparatus 101 sets ObjX that is in contact as a child element of the cursor object 901 using the virtual object management module 623. That is, control is performed so that ObjX follows (moves) the cursor object 901 (three-dimensional model moving means). Since ObjX that has become a child element follows the cursor object 901, the user can freely move the position of ObjX.

  In step S807, the CPU 501 of the information processing apparatus 101 determines whether the first button 402 of the mouse 401 is pressed using the input device control module 602. If it is determined that the first button 402 is pressed, the process proceeds to step S808. If it is determined that the first button 402 has not been pressed and the second button 403 has been pressed, the virtual space generation process ends, and the process proceeds to step S713.

  In step S808, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to obtain the coordinate value in the virtual space of ObjX that has become a child element of the cursor object 901 in step S806 described above. The coordinate value acquired here is used in step S812 described later.

  In step S809, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to check whether ObjX that is a child element of the cursor object 901 is in contact with a virtual object that is different from the ObjX (hereinafter, ObjY). Determine if. ObjX that has become a child element of the cursor object 901 in step S806 follows the cursor object 901 as described above. That is, as a result of the user moving ObjX with the cursor object 901, ObjX may come into contact with ObjY, which is another virtual object. As indicated by 1100 in FIG. 11 described above, in this embodiment, when the virtual object 1002 (ObjX) moved by the cursor object 901 comes into contact with another virtual object 1003 (ObjY), the virtual object 1002 ( ObjX) must be stopped at the position of contact, and the tracking of the cursor object 901 must be terminated. Therefore, in step S809, it is determined whether ObjX that is a child element of the cursor object 901 is in contact with ObjY. Further, when determining contact, it is further determined whether there is ObjY in the movement direction of ObjX. In other words, if ObjX is directed toward ObjY, it is necessary to stop when it comes into contact, but in other cases (when ObjX is moved parallel to a surface with ObjY), it is in contact. There is no need to stop. Therefore, the contact determination is performed in consideration of the moving direction of ObjX. If it is determined that ObjX and ObjY are in contact with each other, the process proceeds to step S810. If it is determined that ObjX and ObjY are not in contact with each other, the virtual space generation process is terminated, and the process proceeds to step S713.

  In step S810, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to identify and display a place where ObjX and ObjY are in contact.

  It is assumed that the user moves ObjX without worrying about contact between virtual objects. For this reason, when ObjX suddenly stops following, the user feels uneasy. Therefore, such a problem is solved by notifying the user that ObjX and ObjY are in contact with each other.

  Also, the user may want to know which part is in contact. For example, in product design, failure to check whether a part is in contact with other parts may lead to failure. Therefore, such a problem is solved by identifying to the user which part of ObjX is in contact with ObjY. An example of this identification display is shown at 1100 in FIG. In 1100 of FIG. 11, since the left surface of the virtual object 1002 (ObjX) and the right surface of the virtual object 1003 (ObjY) are in contact, a balloon is displayed to indicate the contacted portion. In addition to this, the color of the contacted part may be changed, or a sound may be emitted from a not-shown speaker. Any method of notification may be used.

  In step S811, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to separate the child element ObjX from the cursor object 901 from the cursor object 901. In other words, it is not a child element. In step S812, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to substitute the coordinate value acquired in step S808 for ObjX. That is, since ObjX is no longer a child element of the cursor object 901, it is necessary to end the follow-up and stop at the touched position. Therefore, the coordinate value acquired in step S808 is used as the display position of ObjX.

  In step S813, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to change the first button pressing flag from “0” to “1”. The first button pressing flag is a flag indicating that the first button 402 of the mouse 401 is being pressed. The first button pressing flag is used in step S814 to be described later. When the process of step S813 ends, the virtual space generation process ends, and the process proceeds to step S713.

  In this manner, the user can move ObjX by pressing the first button 402 or the second button 403 while the cursor object 901 is in contact with ObjX, and the ObjX can be moved with the first button 402. When the object is moved, if ObjX comes in contact with ObjY, which is another virtual object, ObjX can be stopped at the contacted position. Therefore, an operation closer to the real space can be presented to the user. In addition, when ObjX is moved by the second button 403, Steps S808 to S813 are not executed, and ObjX penetrates ObjY without stopping at the position where ObjX contacts ObjY as before. Can be moved. That is, the user can select a desired moving method according to the situation.

  On the other hand, if it is determined in step S805 that ObjX does not contact the cursor object 901, in step S814, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to set the first button pressed flag. It is determined whether or not “1”. That is, it is determined whether or not the first button 402 is being pressed even after the processing of steps S808 to S813 is executed. If it is determined that the first button pressing flag is “1”, the process proceeds to step S815. If it is determined that the first button pressing flag is “0”, the virtual space generation process is terminated, and the process proceeds to step S713.

  In step S815, the CPU 501 of the information processing apparatus 101 determines whether the second button 403 is further pressed using the input device control module 602. If it is determined that the second button 403 is also pressed, the process proceeds to step S816. If the second button 403 has not been pressed, the virtual space generation process ends, and the process proceeds to step S713.

  In step S <b> 816, the CPU 501 of the information processing apparatus 101 acquires a coordinate value in the virtual space of the cursor object 901 using the virtual object management module 623. In step S817, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to substitute the coordinate value acquired in step S816 as the coordinate value of ObjX. That is, ObjX stopped at the position touched in step S812 described above is called back to the position of the cursor object 901.

  In step S818, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to set ObjX as a child element of the cursor object 901, as in step S806 described above. In this way, even when ObjX is disconnected from the cursor object 901 by contacting ObjY and stopped, the second button 403 can be further pressed while the first button 402 continues to be pressed. , ObjX can be recalled. That is, the follow operation of ObjX can be resumed. As a result, it is possible to reduce the trouble for the user to go to ObjX.

  If it is determined in step S803 that neither the first button 402 nor the second button 403 has been pressed, the information processing apparatus 101 determines in step S819. The CPU 501 uses the virtual object management module 623 to determine whether or not the cursor object 901 has a child element ObjX. That is, since the pressing of the first button 402 has ended, ObjX must be disconnected from the cursor object 901. Therefore, such a determination is performed in step S819. If it is determined that the cursor object 901 has a child element, the process proceeds to step S820. If it is determined that the cursor object 901 has no child element, the process proceeds to step S823.

  In step S820, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to acquire the coordinate value in the virtual space of ObjX following the cursor object 901, as in step S808 described above. In step S821, the CPU 501 of the information processing apparatus 101 separates ObjX from the cursor object 901 using the virtual object management module 623, as in step S811 described above. In step S822, the CPU 501 of the information processing apparatus 101 uses the virtual object management module 623 to substitute the coordinate value acquired in step S820 for the coordinate value of ObjX, as in step S812 described above. By doing so, the follow operation of ObjX is completed.

  In step S823, the CPU 501 of the information processing apparatus 101 sets the first button pressing flag to “0” using the virtual object management module 623. That is, the first button pressing flag is initialized.

  As described above, a virtual space is generated with the virtual object operated by the user through the mouse 401, and prepared for the processing after step S713 described above.

  As described above, when the 3D model in the virtual space operated by the user comes into contact with a 3D model different from the 3D model, the 3D model operated by the user can be stopped. The effect which can show more realistic operation is produced.

  The present invention can be implemented as a system, apparatus, method, program, storage medium, or the like, and can be applied to a system including a plurality of devices. You may apply to the apparatus which consists of one apparatus.

  Note that the present invention includes a software program that implements the functions of the above-described embodiments directly or remotely from a system or apparatus. The present invention also includes a case where the system or the computer of the apparatus is achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a flexible disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, and CD-RW. In addition, there are magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the downloaded key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. In addition, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  Note that the above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

100 MR system 101 Information processing apparatus 102 HMD
103 Marker 201 Transmitter 202 Receiver 301 Infrared Camera 302 Optical Marker 401 Mouse 501 CPU
502 ROM
503 RAM
504 System bus 505 Input controller 506 Video controller 507 Memory controller 508 Communication I / F controller 509 Input device 510 Display 511 External memory 521 Right-eye video camera 522 Left-eye video camera 523 Right-eye display 524 Left-eye display 525 Controller

Claims (9)

An information processing apparatus for presenting to a user a mixed reality image in which a virtual space image that is an image obtained by imaging a virtual space is superimposed on a real space image that is an image obtained by imaging a real space,
A real space image input receiving means for receiving an input of the real space image;
Position and orientation input accepting means for accepting input of the position and orientation of the user presenting the mixed reality image;
A setting unit configured to set the three-dimensional model arranged in the virtual space to follow the movement of the object held by the user in the real space in response to a first instruction from the user;
3D model set by the setting means, and a three-dimensional model moving means for moving so as to follow the movement of the object,
The virtual space image is generated by imaging the virtual space including the three-dimensional model moved by the three-dimensional model moving unit based on at least one of the position and the posture received by the position and posture input receiving unit. Virtual space image generation means for
Mixed reality image generating means for generating the mixed reality image by superimposing the virtual space image generated by the virtual space image generating means on the real space image received by the real space image input receiving means;
The setting means follows the movement of the object set by the setting means when a three-dimensional model moved following the movement of the object contacts a three-dimensional model different from the three-dimensional model. Cancel the setting to
The three-dimensional model moving unit does not cause the moved three-dimensional model to follow the movement of the object held by the user in response to the setting of the setting unit canceling the setting of following the movement of the object. the information processing apparatus which is characterized in that stop at a position come in contact with different three-dimensional model with the three-dimensional model. When the setting means detects a second instruction different from the first instruction, the setting means again causes the three-dimensional model stopped by the three-dimensional model moving means to follow the movement of the object. Set,
The three-dimensional model moving means moves said when examined knowledge of different second instruction from the first instruction, the 3D model obtained by stopping the movement in the three-dimensional model moving means to a position of the object It causes the information processing apparatus according to claim 1, characterized in that makes resume movement to follow the movement of the object. The object is an input device that is communicably connected to the information processing apparatus,
Wherein the position and orientation input accepting unit further accepts an input of the position and orientation of the entering force device,
The three-dimensional model moving means according to claim 1 and feature a user to move the three-dimensional model which is specified based on the position and orientation of the input device, the input of which is accepted by the position and orientation input receiving means or 2. The information processing apparatus according to 2. The 3D model moving unit generates a 3D model in a virtual space based on the position and orientation of the input device that has received an input by the position / orientation input receiving unit, and is in contact with the generated 3D model the information processing apparatus according to claim 3, feature to move the model.   When the three-dimensional model moved by the first instruction from the user comes into contact with a three-dimensional model different from the three-dimensional model, the three-dimensional model moving means moves the three-dimensional model at the touched position. The movement is stopped, and the movement of the three-dimensional model is not stopped when the three-dimensional model moved by the second instruction from the user comes into contact with a three-dimensional model different from the three-dimensional model. The information processing apparatus according to any one of claims 1 to 4.   The position / orientation input accepting unit accepts input of the position and orientation of the user by receiving information transmitted from a sensor connected to be communicable with the information processing apparatus. 6. The information processing apparatus according to any one of 5 above. The information processing apparatus is communicably connected to a head mounted display capable of imaging a real space,
The real space image input receiving means receives the input of the real space image captured by the head mounted display,
The information processing apparatus includes:
A mixed reality image transmitting means for transmitting the mixed reality image generated by the mixed reality image generating means to the head mounted display;
The head mounted display is
The information processing apparatus according to claim 1, further comprising display means for displaying a mixed reality image transmitted from the information processing apparatus. A control method of an information processing apparatus for presenting a mixed reality image in which a virtual space image that is an image obtained by imaging a virtual space is superimposed on a real space image that is an image obtained by imaging a real space to a user,
A real space image input receiving unit of the information processing apparatus for receiving an input of the real space image;
A position / orientation input receiving step in which a position / orientation input receiving unit of the information processing apparatus receives an input of a position and posture of a user presenting the mixed reality image;
A setting step in which the setting unit of the information processing apparatus sets the three-dimensional model arranged in the virtual space to follow the movement of the object held by the user in the real space in accordance with an instruction from the user;
3D model moving means of the information processing apparatus, a three-dimensional model that has been set in the setting step, the three-dimensional model moving step of moving so as to follow the movement of the object,
The virtual space image generation means of the information processing apparatus is based on at least one of the position and the posture of the virtual space including the three-dimensional model moved in the three-dimensional model moving step. A virtual space image generation step of generating the virtual space image by imaging
The mixed reality image generating means of the information processing apparatus superimposes the virtual space image generated in the virtual space image generating step on the real space image received in the real space image input receiving step, and the mixed reality image A mixed reality image generation step for generating
The setting step follows the movement of the object set in the setting step when a three-dimensional model moved following the movement of the object contacts a three-dimensional model different from the three-dimensional model. Cancel the setting to
The three-dimensional model moving step does not cause the moved three-dimensional model to follow the movement of the object held by the user in response to the cancellation of the setting to follow the movement of the object in the setting step. the control method of an information processing apparatus, characterized in that the stop at a position come in contact with different three-dimensional model with the three-dimensional model. An information processing apparatus control method for presenting to a user a mixed reality image obtained by superimposing a virtual space image, which is an image obtained by imaging a virtual space, on a real space image, which is an image obtained by imaging the real space, is executable. A program,
The information processing apparatus;
A real space image input receiving means for receiving an input of the real space image;
Position and orientation input accepting means for accepting input of the position and orientation of the user presenting the mixed reality image;
A setting unit configured to set a three-dimensional model arranged in the virtual space to follow the movement of the object held by the user in the real space in response to an instruction from the user;
3D model set by the setting means, and a three-dimensional model moving means for moving so as to follow the movement of the object,
The virtual space image is generated by imaging the virtual space including the three-dimensional model moved by the three-dimensional model moving unit based on at least one of the position and the posture received by the position and posture input receiving unit. Virtual space image generation means for
Superimposing the virtual space image generated by the virtual space image generation means on the real space image received by the real space image input reception means, and functioning as a mixed reality image generation means for generating the mixed reality image;
The setting means follows the movement of the object set by the setting means when a three-dimensional model moved following the movement of the object contacts a three-dimensional model different from the three-dimensional model. Cancel the setting to
The three-dimensional model moving unit does not cause the moved three-dimensional model to follow the movement of the object held by the user in response to the setting of the setting unit canceling the setting of following the movement of the object. a program characterized by stop at a position come in contact with different three-dimensional model with the three-dimensional model.

Images