JP3449937B2 - Panorama image creation method, surrounding situation transmission method using panorama image, and recording medium recording these methods - Google Patents

Description

Detailed Description of the Invention

[0001]

The present invention relates to a user who has participated in a three-dimensional virtual space and displays his or her surroundings in the real space by using a panoramic image that is updated in real time in the virtual space so that other users can see the surrounding situation. It is about the technology to convey to.

[0002]

2. Description of the Related Art There is a system in which a three-dimensional virtual space is described by computer graphics, and a user operates an avatar as his or her own altercation in the space to meet other users and communicate with them. Conventionally, in such a system, the user has had to use words or characters to convey the situation in the real space in which he / she is.

Also, a system for drawing a graphic in a virtual space, or taking a face image of a user using a video camera and attaching it to the face of an avatar to reflect the facial expression in the virtual space. There is. In such a system, a user uses a picture to convey the situation of the real space in which he / she is present, or displays it to the other party through a video image for a low resolution face image.

[0004]

However, in the conventional cyber communication in the three-dimensional virtual space, the communication may not engage because the surrounding situation of the user in the real space cannot be known.
I don't have the chance to talk because there is little information about other users. For example, when communicating with people on the other side of the earth in the virtual space through the Internet, even though various elements such as season, climate, and time are different in the real space, they are in the same virtual space, Sometimes I misunderstand that the other person is in the same situation as myself, and proceed with the story. Also, because you can communicate only with voice and text, and you can only send images with limited resolution, even if you try to convey the situation of your own real space to the other party through the virtual space, you can express it in words. I have no choice but to. Even if it can be immediately conveyed to the other person if it is shown in a video, it must be communicated to the other person using expressions in a messy language. In this way, despite the three-dimensional virtual space display that can handle video information, the surrounding situation of the user in the real space that I want to convey is
As with language-based communication methods, there was the problem of having to communicate in different languages.

The present invention has been made in order to solve the above-mentioned problems, and more information can be obtained when communicating in a three-dimensional virtual space, the communication is activated, and the real time of each other is obtained. It is an object to provide an image generation method and a situation communication method for realizing smooth communication while showing a situation.

[0006]

The present invention solves the above problems by the means listed below.

[0007] One of the means is to take a picture with a video camera.
Import the image with position information and angle information,
Based on the information, measuring the amount of movement from the reference position, the angle
The image calculated according to the degree information and the movement amount
A panorama image to be pasted on a tubular object is created from the position coordinates of the area and the image area, and the panorama image is created.
The image is updated even if it exceeds a certain time in the image creation stage.
The panorama image area that does not
The display image disappears so that new images are constantly displayed.
A method of creating a panoramic image, characterized by creating a real-time panoramic image displaying image information .

Alternatively, the position photographed by the video camera
Information and angle information captures are marked with images, to determine the input first image areas in the reference image, or the position
Newly determined reference image an image region that is input when the amount of movement to measure the movement amount of the reference picture image or colleagues exceeds a certain value on the basis of information about the input image area from the angle information The rotation of the optical axis of the video camera is corrected horizontally, and the corrected image area is projectively converted into a cylindrical object according to the rotation of the optical axis in the horizontal direction and the vertical direction. With respect to the reference image, and the above processing is sequentially performed on the image areas that are sequentially input to create a panoramic image to be attached to a cylindrical object.
Is not updated even if it exceeds a certain time in the creation stage of
The panoramic image area will be reduced in image density after a certain time.
Display image disappears, new image information is always displayed.
A method for creating a panoramic image, characterized by creating a real-time panoramic image displaying information .

[0009]

[0010] Other means, in a system that takes other users communicate user moves his alter ego in a three-dimensional virtual space, the panoramic image generated by the panorama image generation method above mentioned in the real space A surrounding situation transmission method using a panoramic image, characterized by displaying as a surrounding situation image of the user.

[0011]

Furthermore, a recording medium recording a panoramic image creating method, characterized in that a program for causing a computer to execute the procedure in the panoramic image creating method described above is recorded in a recording medium readable by the computer. is there.

Alternatively, a program for causing a computer to execute the procedure in the above-described method of transmitting a surrounding situation using a panoramic image is recorded in a computer-readable recording medium, and the surroundings using the panoramic image are characterized. It is a recording medium in which a situation communication method is recorded.

According to the present invention, a user in a real space wears a video camera capable of freely changing the optical axis direction in the real space, and based on an image taken by the video camera,
A panoramic image with sufficient resolution is created in real time and sent to the virtual space of another user. Then, on the other user side, the panoramic image is pasted and displayed around the avatar of the image transmission user in the virtual space, so that the other user knows the real space situation of the user of the avatar being displayed. To be able to. The present invention is thus characterized in that the situation in the real space is transmitted to other users in the virtual space.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION Below, an example of an embodiment of a panoramic image creating method and a surrounding condition transmitting method using a panoramic image of the present invention will be shown.

[Embodiment 1] Embodiment 1 shows an embodiment of a panoramic image creating method.

FIG. 1 shows an example of a video camera used by a caller user. Reference numeral 11 is a video camera, 12 is a position for measuring the optical axis direction of the video camera, and an angle sensor for measuring the position of the video camera, and 13 is a computer. The computer 13 captures the image from the video camera 11 together with the measured position and angle information of the video camera.

Here, the positional relationship of the objects shown in the image deviates depending on the movement of the video camera 11.
It is necessary to compose this into a single image, but the conventional panoramic image generation method generates a panoramic image based on a video image that has been shot in advance, or even if it is generated in real time, it has a certain orientation. Or, a panoramic image was generated for a certain range of movement of the video camera. This method presents a panoramic image generation and display method when the position of the video camera changes or the optical axis direction of the video camera changes in real time.

FIG. 2 is a flow chart showing a panoramic image generation and display method according to this embodiment.

As shown in FIG. 3A, the user in the real space can freely move in the space. Reference numeral 21 represents an image transmitting user before moving, and 22 represents an image transmitting user after moving. Here, the image transmitting users 21 and 22 place the video camera 11 with the position and angle sensor shown in FIG. 1 on their heads, for example, and carry the portable computer 13.

The position (X, Y, Z) and angle (α, β, γ) information as shown in FIG. 3B is added by the video camera 11 and the position / angle sensor 12 of FIG. 3A. It is assumed that the frame images of are captured by the computer 13 in the order of f1, f2, f3 and f4. Here, the angle is α with respect to the vertical axis, β is with respect to the horizontal axis, and γ is with respect to the optical axis of the video camera.

First, the reference image is determined. If the frame image (f1) is the first input image, it becomes the reference image. If the reference image has already been determined, the movement amount of the camera from the camera position of the reference image is measured from the position (X, Y, Z). If the amount of movement of the camera is too large, it will not be possible to match the images in the panoramic image being generated, so if the amount of movement exceeds a certain reference value,
A new panoramic image is generated using the image at that position as a reference image (reference plane).

Next, when the reference image (reference plane) is determined, a panoramic image generation process is performed using the angles (α, β, γ) of the frame image (f1).

First, the image is rotated so that the angle γ indicates 0 degree, and image adjustment for correcting the horizontal of the image is performed.

Next, the frame image (f1) is converted into the angle α.
Perform projective transformation on. At this time, in order to generate a panoramic image that can be attached to a cylindrical octahedron object as shown in FIG. 4A, 0 degrees relative to the reference image is set to 0 degree as the reference plane, 45 degrees, 90 degrees, and 135 degrees. , 180 degrees, 225
An image parallel to the surfaces of 270 degrees and 315 degrees is used as a quasi-reference plane (FIG. 4 (2)).

When the angle α is 337.5 degrees to 22.5 degrees, it is on the reference plane, and when the angle α is 22.5 degrees to 67.5 degrees, it is on the quasi-reference plane of 45 degrees. Sometimes 90
To the quasi-reference plane of 112.5 degrees to 157.5 degrees to the quasi-reference plane of 135 degrees from 157.5 degrees to 202.5 degrees
When the degree is 180 degrees, the quasi-reference plane of 180 degrees, from 202.5 degrees to 2
At 47.5 degrees, the 225 degree quasi-reference plane is 247.5 degrees.
When the angle is from 33 degrees to 337.5 degrees, the projective transformation is performed on the quasi-reference plane of 315 degrees.

Next, the projective transformation is performed using the angle β so that the image becomes vertical.

Next, matching with the reference image is performed using corresponding points of the image. As the corresponding points, feature objects such as landmarks by edge detection can be used. In matching, if necessary due to the positional relationship with the reference image, the image is enlarged / reduced.

The above-described processing is sequentially performed in the same manner for the frame images f2, f3, and f4 that are input one after another to create a cylindrical panoramic image.

The portion where the frame image newly attached to the panorama image overlaps with the frame image previously attached is updated to the latest frame image. The latest panoramic image created in this manner can be taken out and displayed at any time, or can be used as in the embodiment described later.

[Embodiment 2] Embodiment 2 is more suitable for transmitting the panoramic image generated in Embodiment 1 in real time in real time in cyber communication or the like. 7 illustrates an example embodiment of a panoramic image generation method.

Based on the image newly input for each frame from the video camera and the position and angle information of the video camera input from the sensor, the corresponding area on the panoramic image generated in the first embodiment is Update the image. Further, for the image of the area that is not updated for a certain period of time, after a certain period of time, the density of the image of the region is reduced and finally erased. That is, the updated and displayed panoramic image becomes an image of the user's surroundings captured by the user's camera within a certain period of time.

[Embodiment 3] Embodiment 3 is a method for transmitting a real-time situation of oneself in real time in cyber communication or the like using the panoramic image generated in Embodiment 1 or 2. A form example is shown.

FIG. 5 shows a system diagram. The image transmitting user shown by 51 has a video camera 11 with a position and angle sensor 12 and a portable computer 13, and the first embodiment
Alternatively, a panoramic image is generated by the method shown in 2.

As shown in FIG. 6, the difference between the optical axis direction of the video camera at the time when the video camera starts shooting and the direction of the user's front sight line is defined as the initial relative angle α, and the image at that time is the reference position image 61. And Then, a panoramic image is generated from the captured image by the method described in the first or second embodiment, and is transmitted via the transmission computer 54 or the PH.
S or the like is used to send to the server shown at 52 connected to the network 56. The server 52 sends the sent image together with the position information of the reference position image and the initial relative angle information to 5
The image is sent to the receiving computer 55 of the image receiving user (client) shown in FIG.

As shown in FIG. 7, the receiving computer 5
In the three-dimensional virtual space of 5, the cylindrical object indicated by 71 is displayed around the caller avatar of 72. A cylindrical image 71 is a panoramic image created based on the initial relative angle and the reference position image 61, with the front direction of the avatar of the user who is generating the panoramic image entering the virtual space as a reference. Paste it on the object. That is, the created panoramic image is displayed around the avatar 72 of the image transmission user. Other users in the same virtual space know the surroundings of the image transmitting user by looking at the displayed image. The image displayed on the cylindrical object around the avatar matches the positional relationship of the user's surroundings in the real space.

[Embodiment 4] In Embodiment 4, in the panoramic image display method in the virtual space shown in Embodiment 3, only the area on the panoramic image that is updated for each frame when sending an image. Is sent to the client together with the position and angle information, and the receiving computer 55 of the client reconstructs the panoramic image based on the sent image and the position and angle information, and maps it to the corresponding pasting position. By doing so, it is possible to display and update an image with a large resolution without imposing a heavy load on the network 56.

A computer-readable recording medium such as an FD (floppy disk) or a program for executing the processing shown in FIG. 2 or the processing described with reference to FIG. MO, R
It can be recorded and provided on an OM, a memory card, a CD, a DVD, a removable disk, etc., and distributed. In the system of FIG. 5, the programs required by the transmission side computer and the reception side computer may be recorded in the same recording medium, and the respective programs may be activated. The programs to be used may be recorded in different recording media.

[0039]

As described above, by capturing the surroundings of the user in the real space with a video camera, constructing the image as a panoramic image, and displaying it around the avatar in the three-dimensional virtual space, It is possible to grasp the surrounding situation in the real space. As a result, when communicating in a three-dimensional virtual space, more information can be obtained, the communication is activated, and there is an effect of realizing smooth communication while showing the situation in each other's real space.

[Brief description of drawings]

FIG. 1 is a diagram showing an example of a video camera of an image input unit used in an embodiment of the present invention and a computer unit for taking an image and making a panorama.

FIG. 2 is a flowchart showing a processing flow of real-time panoramic image generation according to the first embodiment of the present invention.

[3] (1) is a diagram showing the movement of the user in the embodiment 1, (2) is in correspondence with the deviation of the position and angle due to the movement, is a diagram of the image by spatial arrangement captured.

4 (1) and (2) are diagrams showing a cylindrical octahedral object for pasting a panoramic image generated according to the first embodiment.

FIG. 5 is a diagram showing a network system for a surrounding condition transmitting system using a panoramic image, which enables the surrounding condition transmitting method according to Embodiments 3 and 4 of the present invention.

FIG. 6 is an explanatory diagram of a reference image determination method at the time of displaying a panoramic image according to the third embodiment.

FIG. 7 is an explanatory diagram of a method of pasting a panoramic image around an avatar in a three-dimensional virtual space according to the third embodiment.

[Explanation of symbols]

11 ... Video camera 12 ... Position and angle sensor 13 ... Computer 21 ... Image sending user before moving 22 ... Image sending user after moving 51 ... Image transmission user 52 ... server 53 ... Image receiving user 54 ... Sending computer 55 ... Receiving computer 56 ... Network 61 ... Reference position image 71 ... Object for pasting panoramic image 72 ... Originator avatar in virtual space

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Susumu Ichinose 3-19-2 Nishishinjuku, Shinjuku-ku, Tokyo Inside Nippon Telegraph and Telephone Corporation (72) Inventor Aiko Kitagawa 3-19-3 Nishishinjuku, Shinjuku-ku, Tokyo No. 2 Nihon Telegraph and Telephone Corporation (56) References Japanese Patent Laid-Open No. 10-188029 (JP, A) Japanese Patent Laid-Open No. 10-187929 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB) Name) G06T

Claims (5)

(57) [Claims] 1. Position information and corners taken by a video camera
Download the images degree information is attached, based on the position information, the movement amount from the reference position is measured and calculated in accordance with the angle information and the amount of movement, before
A panoramic image to be pasted on a cylindrical object is created from the position coordinates of the image area and the image area , and a certain time is exceeded in the panoramic image creating step.
The panorama image area that is not updated even after a certain time
By decreasing the density of the
Real-time panoramic image displaying new image information on
A method of creating a panoramic image, which is characterized by: 2. Position information and corners taken by a video camera
Download the images degree information with the first input image area determined in the reference image, or the amount of movement to determine the movement amount of the reference picture image or et on the basis of said position information exceeds a predetermined value newly determined reference image an image area that is entered when the, bidet for the angular information from the inputted image area
Oh, the rotation of the optical axis of the camera is corrected horizontally, and the corrected image area is projectively converted into a cylindrical object according to the rotation of the optical axis in the horizontal direction and the vertical direction. the matching is performed with the reference image, to create a panoramic image to be pasted to the cylindrical object by sequentially executing the image area to be sequentially inputted to the processing described above, during the preparation of the panoramic image, exceeds a predetermined time
The panorama image area that is not updated even after a certain time
By decreasing the density of the
Real-time panoramic image displaying new image information on
A method of creating a panoramic image, which is characterized by: 3. A system for a user in a three-dimensional virtual space take other users communicate by moving his alter ego, generated in 請 Motomeko 1 or 2, wherein the panorama image generating method
A method of transmitting a surrounding situation using a panoramic image, characterized by displaying the panoramic image as a surrounding situation image of the user in a real space. 4. A panoramic image creating method, characterized in that a program for causing a computer to execute the procedure in the panoramic image creating method according to claim 1 or 2 is recorded in a computer-readable recording medium. Recording medium. 5. A panoramic image, characterized in that a program for causing a computer to execute the procedure in the method of transmitting a surrounding situation using a panoramic image according to claim 3 is recorded on a computer-readable recording medium. A recording medium that records the method of transmitting surrounding conditions used.