JP4032403B2 - Virtual reality space sharing system, virtual reality space sharing method, client, data transmission method, and recording medium on which data transmission program is recorded - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a virtual reality space sharing system, a virtual reality space sharing method, and a recording medium in which a plurality of operators share a virtual reality space via a network , and in particular, moves an individual object of each operator in the virtual reality space. The present invention relates to a virtual reality space sharing system, a virtual reality space sharing method, and a recording medium .
[0002]
[Prior art]
Conventionally, there is a virtual reality space sharing system that allows a plurality of operators to share a virtual reality space and freely move an individual object of each operator in the space to communicate between the operators ( For example, JP-A-9-081782). In such a system, when an operator wants to change the facial expression or movement of an object of his / her own part, a change instruction is issued on the client terminal. Then, the server side that has received this instruction issues an instruction to change the facial expression and movement of the object to the other client terminal, whereby the other client terminal displays the facial expression and movement of the corresponding object on each screen. To change.
[0003]
[Problems to be solved by the invention]
However, in such a conventional virtual reality space sharing system, the operator uses a mouse or the like on the client terminal to select and instruct the desired facial expression, so the operation is troublesome, It was not possible to change the expression of objects naturally.
[0004]
The present invention has been made in view of the above points, and a virtual reality space sharing system and a virtual reality space that can naturally change the expression of an object as the operator desires without performing a complicated operation. It is an object to provide a sharing method and a recording medium .
[0005]
[Means for Solving the Problems]
In the present invention, in order to solve the above-described problem, in a virtual reality space sharing system in which a plurality of operators can share a virtual reality space via a network, and a separate object of each operator can be moved in the virtual reality space. A server that provides the virtual reality space, and a first client and a second client that receive the virtual reality space from the server via the network, wherein the first and second clients are: , Display means for displaying an object of each operator in the virtual reality space, a camera for photographing the face of the operator of the client, and detecting a motion vector of a characteristic portion from image data photographed by the camera Motion vector detecting means, and motion vectors and facial expressions obtained by the motion vector detecting means Only when it is determined that the facial expression of the operator of the client, the facial expression judgment means for judging the change of the facial expression, and the facial expression judgment means have changed the facial expression by comparing the use data, Expression data transmitting means for transmitting facial expression data of the client operator's facial expression to the server side and changing an image of a separate object of the client operator on the transmission side, and a screen in accordance with a command from the server Image change means for changing the image of the object above, and the server changes the expression of the object corresponding to the operator on the screen of each client based on the transmitted expression data There is provided a virtual reality space sharing system characterized by comprising facial expression change command means for performing the above.
[0006]
In such a virtual reality space sharing system, first, a virtual reality space is provided to each client by a server via a network. On each client side, the motion vector detection means detects the motion vector of the characteristic portion from the image data captured by the camera that captures the face of the client operator, and the facial expression determination means detects the detected motion vector and facial expression. The facial expression of the client operator is judged by comparing the judgment data . Then, the expression data transmission means transmits the determined facial expression data of the client operator to the server side.
[0007]
In the server, the facial expression change commanding unit issues a command to change the facial expression of the object corresponding to the operator on the screen of each client based on the transmitted facial expression data. In each client that receives this, the image changing means changes the image of the object on its own screen in response to a command from the server.
[0008]
Further, according to the present invention, in a virtual reality space sharing method in which a plurality of operators can share a virtual reality space via a network, and an individual object of each operator can be moved in the virtual reality space. The first client and the second client that receive the virtual reality space from the server display the objects of each operator in the virtual reality space and photograph the face of the operator of the client with the camera. And detecting the motion vector of the characteristic portion from the image data photographed by the camera, comparing the obtained motion vector with the facial expression determination data, and comparing the facial expression of the operator of the client device with the facial expression Only when it is determined that there has been a change in the facial expression, a change in facial expression of the client operator's face is determined. The chromatography data by changing the image of an object on the client of the operator's alter ego sender and sends to the server, to change the image of the object on the screen in response to a command from the server, the virtual reality space server, on the basis of the data of the transmitted expression, virtual reality space sharing method is provided which is characterized in that a command to change the facial expression of the object corresponding to the operator on the screen of the client to provide The
[0011]
Furthermore, according to the present invention, in the virtual reality space sharing system in which a plurality of operators can share the virtual reality space through the network and the objects of each operator can be moved in the virtual reality space. Displaying an object of each operator in the virtual reality space on a first client and a second client that receive the provision of the virtual reality space from a server, and displaying the face of the operator of the client A step of photographing with a camera; a step of detecting a motion vector of a characteristic portion from image data photographed by the camera; and a comparison between the obtained motion vector and expression determination data, A facial expression, a step of judging a change in the facial expression, and a change in the facial expression. Only if it is, a step of changing the image of an object of a client of the operator's alter ego sender and sends the data of the facial expression of the operator of the client to the server, depending on the instruction from the server And changing the image of the object on the screen to the server that provides the virtual reality space to the operator on the screen of each client based on the transmitted facial expression data. A computer-readable recording medium recording a program for executing a process including a step of giving a command to change the expression of a corresponding object is provided.
[0014]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a concept of functions of the virtual reality space sharing system of this embodiment. A server terminal 2 and a large number of client terminals 3 and 4 are connected to the network 1. Each client terminal 3, 4 is provided with facial expression detection means 3 a, 4 a comprising a CCD camera and facial expression detection circuit. These facial expression detection means 3a and 4a detect whether the facial expression of the operators 3A and 4A of the client terminals 3 and 4 is, for example, a smile, an angry face, a crying face, or a surprised face. Expression data detected by the expression detection means 3a, 4a is sent to the server by the expression data transmission means 3b, 4b. However, in this case, it is preferable to transmit facial expression data only when the facial expressions of the operators 3A and 4A change.
[0015]
On the server terminal 2 side, the facial expression change command means 2a instructs to change the facial expression of the object corresponding to the operator on the screens 3d and 4d of the client terminals 3 and 4 based on the transmitted facial expression data. . In the client terminals 3 and 4 that have received this, the image changing means 3c and 4c change the image of the object on their own screens 3d and 4d in response to a command from the server. However, it is preferable that the client terminal that has sent facial expression data to the server terminal 2 changes the display on its own screen without waiting for a command from the server terminal 2.
[0016]
FIG. 2 is a block diagram showing a configuration example of the virtual reality space sharing system of this embodiment. Here, for example, two server terminals 12 and 13 are connected to a network 11 such as the Internet. Storage devices 14 and 15 such as hard disk devices are connected to the server terminals 12 and 13, respectively. The storage devices 14 and 15 store the virtual reality space sharing system program, and can provide the virtual reality space to the client terminals 16, 17, and 18 via the network 11.
[0017]
The virtual reality space is displayed as an image on a monitor, which will be described later, of each client terminal 16, 17, 18. In this virtual reality space, an object of the operator who operates each of the client terminals 16, 17 and 18 is displayed, and by operating a keyboard, a mouse, etc., various parts of the body and facial expressions can be freely set in the space. Can be moved to. In addition to the object of the operator, the background building can also be moved.
[0018]
A plurality of service provider terminals 19 and 20 are connected to the network 11. The service provider terminals 19 and 20 are provided with a virtual reality space via the network 11 and can provide predetermined services to the client terminals 16, 17 and 18.
[0019]
Next, a specific configuration of each terminal will be described.
FIG. 3 is a block diagram showing a specific configuration of the server terminal 12. A CPU 21, ROM 22, RAM 23, communication device 24, interface 25, and the like are connected to the bus 30. The CPU 21 controls the operation of the entire server terminal 12 according to a program built in the ROM 22. The RAM 23 appropriately stores data, programs, and the like necessary for the CPU 21 to execute various processes. The communication device 24 exchanges data with the network 11.
[0020]
The storage device 14, display device 26, microphone 27, speaker 28, and input device 29 are connected to the interface 25. The storage device 14 is a hard disk, an optical disk, a magneto-optical disk, or the like, and stores programs and data necessary for providing a virtual reality space. The display device 26 includes a CRT, an LCD, and the like, and displays an image of the virtual reality space provided by the server terminal 12.
[0021]
Necessary audio signals are input from the microphone 27 and supplied to the client terminals 16, 17, 18 and the service provider terminals 19, 20. On the other hand, audio signals supplied from the client terminals 16, 17, 18 and the service provider terminals 19, 20 are output from the speaker 28.
[0022]
The input device 29 includes a mouse 29a and a keyboard 29b. Necessary commands are issued by the mouse 29a and the keyboard 29b.
Since the configuration of the server terminal 13 is almost the same as that of the server terminal 12, the description thereof is omitted here.
[0023]
FIG. 4 is a block diagram showing a specific configuration of the client terminal 16. A CPU 31, ROM 32, RAM 33, communication device 34, interface 35, etc. are connected to the bus 43. The CPU 31 controls the overall operation of the client terminal 16 according to a program built in the ROM 32. The ROM 32 stores image data of the face of an operator's alternation object, which will be described later. The RAM 33 appropriately stores data, programs, and the like necessary for the CPU 31 to execute various processes. The communication device 34 exchanges data with the network 11.
[0024]
A display device 36, a microphone 37, a speaker 38, a storage device 39, and an input device 40 are connected to the interface 35. The display device 36 includes a CRT, an LCD, and the like, and displays a CG (computer graphic) three-dimensional image and an image taken by a normal video camera or the like. A necessary audio signal is input from the microphone 37 and supplied to the server terminals 12 and 13 via the network 11. On the other hand, the audio signal supplied from the server terminals 12 and 13 is output from the speaker 38. The storage device 39 is a hard disk, an optical disk, a magneto-optical disk, or the like, and stores programs and data necessary for an operation of moving each object in the virtual reality space.
[0025]
The input device 40 includes a mouse 40a and a keyboard 40b. The mouse 40a is used when a predetermined position is designated. The keyboard 40b is used when inputting predetermined characters and symbols. The input device 40 includes a viewpoint input device 40c and a movement input device 40d. The viewpoint input device 40c is used when inputting a viewpoint in the virtual reality space of an object of the operator of the client terminal 12. Thereby, the viewpoint of the object can be moved up, down, left, and right, and the image ahead of the line of sight of the object can be zoomed. The movement input device 40d is used when the object is moved back and forth and left and right.
[0026]
In addition, a CCD camera 41 and a facial expression detection circuit 42 are connected to the interface 35. The CCD camera 41 always captures the face of the operator of the client terminal 16. The photographed video signal is taken into the facial expression detection circuit 42 at regular intervals. The facial expression detection circuit 42 detects the facial expression of the operator based on the video signal by a procedure described later, and sends the result to the server side.
[0027]
Note that the configuration of the client terminals 17 and 18 is almost the same as that of the client terminal 16, and thus the description thereof is omitted here.
Since the service provider terminals 19 and 20 have substantially the same configuration as the client terminal 16 except for the CCD camera 41 and the expression detection circuit 42, the description thereof is omitted here.
[0028]
Next, a specific configuration of the facial expression detection circuit 42 will be described.
FIG. 5 is a block diagram showing a specific configuration of the facial expression detection circuit 42. The low pass filter (LPF) 421 removes the high frequency component of the video signal input from the CCD camera 41. Since the human face has pores, fine wrinkles, etc., the video signal from the CCD camera 41 contains a large amount of high frequency components. For this reason, the low pass filter 421 removes this high frequency component. The edge detection circuit 422 converts the analog signal output from the low-pass filter 421 into a digital signal, performs edge detection from the digital signal, and outputs video data with only edges.
[0029]
The pattern matching circuit 423 reads out expressionless face data as basic data of the operator's face stored in advance in the memory 424, and performs pattern matching in comparison with the data from the edge detection circuit 422. Adjust the position and size of the face taken. That is, while using the virtual reality space, the operator moves the face position up and down, left and right, or back and forth, and the face coordinate position and size of the basic data differ. This makes it impossible to accurately capture the current facial expression. Therefore, the pattern matching circuit 423 compares the data for one frame from the edge detection circuit 422 with the basic data, and how much the photographed operator's face moves back and forth, left and right, and up and down with respect to the basic data. Detect whether or not The operator's face data photographed based on the movement amount is corrected to the same conditions as the basic data.
[0030]
By the way, the memory 424 stores, as basic data, data of an operator's expressionless face, but also stores position data of characteristic portions that characterize facial expressions.
[0031]
FIG. 6 is a diagram showing an example of basic data of the operator's face stored in the memory 424. The basic face data 51 is stored in the memory 424. Further, as characteristic portions characterizing facial expressions, for example, left eyebrow left end (point P1), left eyebrow right end (point P2), mouth corner left (point P3), mouth corner right (point P4), mouth center upper end (point P5) ), The positions of the right eye corner (point P6) and the left eye corner (point P7). The points P1 to P7 are stored, for example, in XY coordinates on the frame.
[0032]
Returning to FIG. 5, the motion vector detection circuit 425 compares the face data of the operator whose size and position are adjusted by the pattern matching circuit 423 with the basic data 51 of the memory 424, and It is detected how much the points corresponding to the points P1 to P7 shown in FIG. 6 have moved.
[0033]
FIG. 7 is a diagram showing an example of operator's face data imaged by the CCD camera 41. Here, an example of a smile state is shown. The operator's face data 52 is detected by the motion vector detection circuit 425 by the left eyebrow left end (point P11), the left eyebrow right end (point P12), the mouth corner left (point P13), the mouth corner right (point P14), and the mouth center upper end (point P14). Point P15), right eye corner (point P16), and left eye corner (point P17) are detected and compared with the corresponding points P1 to P7 on the basic data 51, respectively. Based on these comparisons, the motion vector detection circuit 425 detects a motion vector indicating how much each of the points P11 to P17 has moved from the corresponding points P1 to P7 of the basic data. This is accomplished by performing block matching of a block containing several pixels in the vicinity containing each point.
[0034]
Returning to FIG. 5, the facial expression determination circuit 426 compares the motion vector obtained by the motion vector detection circuit 425 with the facial expression determination data in the facial expression table 427 to determine the facial expression of the current operator.
[0035]
FIG. 8 is a diagram showing an example of expression determination data in the expression table 427 for determining the facial expression. As shown in the figure, the facial expression judgment data 53 includes “smiling face”, “angry face”, “crying face”, and “surprise face” as facial expressions to be judged. It is given in a moderate range. The facial expression judgment circuit 426 has the facial expression that most closely matches the motion vector of each characteristic part obtained by the motion vector detection circuit 425 in each facial expression of the facial expression judgment data 53 as the facial expression of the current operator. to decide. That is, if it is the data 52 of FIG.
[0036]
The facial expression determination circuit 426 that has determined the facial expression in this way outputs the facial expression code indicating the facial expression to the interface 25 shown in FIG. 4 and sends it to the CPU 31 only when the facial expression determined this time is different from the previous determination. When the facial expression code of its own client terminal is output, the CPU 31 reads the image data of the object corresponding to the facial expression code from the ROM 32 and causes the display device 36 to display it. In this case, the facial expression image of the object is stored in the ROM 32 in accordance with each facial expression code, and the corresponding facial expression image is read from the ROM 32 each time the facial expression code is output. It may be pasted and displayed on the part, or the operation of the face and the whole body corresponding to the expression code is stored in the ROM 32. For example, in the case of “crying face”, the operation of wiping tears is performed. The object may be moved according to the expression code.
[0037]
Simultaneously with the display of such a screen, the CPU 31 sends a display code to the server terminal 12 via the communication device 34 and the network 11. Receiving this, the server terminal 12 sends the expression code and the ID of the client that has output the display code to the other client terminals 17 and 18 via the network 11.
[0038]
Receiving the facial expression code, the client terminals 17 and 18 read the facial expression code and the object corresponding to the ID of the client terminal 16 from the ROM and display them on each display device.
[0039]
FIG. 9 is a flowchart showing a processing procedure on the client terminal side regarding the expression code.
[S1] It is determined whether a facial expression code is output from the facial expression detection circuit 42 of its own terminal. If it is output, the process proceeds to step S2, and if not, the process proceeds to step S4.
[S2] The expression of the object on the screen of the display device 36 is switched to the expression corresponding to the newly output expression code. However, if the operator's own client terminal is set to a mode in which only the scenery of the virtual reality space from the operator's viewpoint is displayed without displaying the operator's alternate object, of course, There is no switching.
[S3] The facial expression code is output to the server terminal 12 side.
[S4] It is determined whether the expression code and the client ID are sent from the server terminal 12. If sent, the process proceeds to step S5, and if not sent, this flowchart ends.
[S5] The facial expression of the alternate object of the client terminal corresponding to the ID sent from the server terminal 12 is switched to the facial expression corresponding to the facial expression code sent simultaneously.
[0040]
FIG. 10 is a flowchart showing a processing procedure on the server terminal side regarding the facial expression code.
[S11] It is determined whether or not an expression code is input from the client terminal side. If it is input, the process proceeds to step S12. If not input, the process of the flowchart ends.
[S12] The inputted facial expression code is transmitted to another client terminal.
[0041]
As described above, in this embodiment, the facial expressions of the operators of the client terminals 16, 17, and 18 are photographed by the CCD camera 41 or the like, and the individual of each operator in the virtual reality space is changed according to the change of the facial expressions. Since the facial expression of the object is changed, it is possible to naturally change the facial expression of the object as the operator intends without performing a complicated operation. Therefore, more realistic communication can be achieved.
[0042]
In this embodiment, when a facial expression code is sent from the client terminal, the server terminal side sends only the facial expression code to the other client terminal, and corresponds to the facial expression code sent on the client terminal side that received it. The image is read from its own ROM 32 and displayed, but all display data may be transmitted from the server terminal to the client terminal.
[0043]
【The invention's effect】
As described above, in the present invention, the facial expression of the operator of each client is detected, and the object image on the screen of each client is changed according to the detected facial expression. Without doing so, the expression of the object can be naturally changed as the operator wishes. Therefore, more realistic communication can be achieved.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a concept of functions of a virtual reality space sharing system of the present embodiment.
FIG. 2 is a block diagram showing a configuration example of a virtual reality space sharing system of the present embodiment.
FIG. 3 is a block diagram showing a specific configuration of a server terminal.
FIG. 4 is a block diagram showing a specific configuration of a client terminal.
FIG. 5 is a block diagram showing a specific configuration of a facial expression detection circuit.
FIG. 6 is a diagram illustrating an example of basic data of an operator's face stored in a memory.
FIG. 7 is a diagram showing an example of operator's face data imaged by a CCD camera.
FIG. 8 is a diagram illustrating an example of expression determination data in an expression table for determining a facial expression.
FIG. 9 is a flowchart showing a processing procedure on the client terminal side regarding an expression code.
FIG. 10 is a flowchart showing a processing procedure on the server terminal side regarding an expression code.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Network, 2 ... Server terminal, 2a ... Expression change command means, 3 ... Client terminal, 3A ... Operator, 3a ... Expression detection means, 3b ... Expression data transmission means, 3c ... Image Change means, 3d... Monitor, 11... Network, 12, 13... Server terminal, 14, 15... Storage device, 16, 17, 18. …… CCD camera, 42 …… Expression detection circuit.

Claims (5)

In a virtual reality space sharing system that allows multiple operators to share a virtual reality space via a network and to move each operator's alternate object in the virtual reality space,
A server for providing the virtual reality space;
A first client and a second client that receive provision of the virtual reality space from the server via the network;
The first and second clients are:
Display means for displaying an object of each operator in the virtual reality space;
A camera for photographing the face of the client operator;
Motion vector detection means for detecting a motion vector of a characteristic portion from image data photographed by the camera;
Comparing the motion vector obtained by the motion vector detecting means with the expression judging data to determine the facial expression of the operator of the client and the facial expression judging means for judging the change in the facial expression;
Only when it is determined that the facial expression has been changed by the facial expression determination means, the facial expression data of the client operator's face is sent to the server side, and an image of a duplicate client operator's operator object image is transmitted. Facial expression data transmission means to be changed ;
Image changing means for changing the image of the object on the screen in response to a command from the server ;
The server
Facial expression change command means for giving a command to change the facial expression of the object corresponding to the operator on the screen of each client based on the transmitted facial expression data.
Virtual reality space sharing system characterized by that .   The facial expression data transmission means transmits an facial expression code indicating the facial expression, the facial expression change command means transmits the received facial expression code to each client side, and the image change means of each client receives the reception 2. The virtual reality space sharing system according to claim 1, wherein image data of a facial expression corresponding to the facial expression code is read from a memory and displayed on the screen. 2. The facial expression data transmitting unit is configured to transmit an expression code indicating the facial expression only when the facial expression determining unit determines that the facial expression has changed . Virtual reality space sharing system. In a virtual reality space sharing method in which a plurality of operators can share a virtual reality space via a network and each operator's alternate object can be moved in the virtual reality space.
A first client and a second client that receive provision of the virtual reality space from a server via the network,
Displaying an alternate object of each operator in the virtual reality space;
Photograph the face of the operator of the client with a camera,
Detecting a motion vector of a feature portion from image data captured by the camera;
Compare the obtained motion vector and facial expression determination data to determine the facial expression of the operator of the client and the change in the facial expression,
Only when it is determined that there is a change in the facial expression, the facial expression data of the client operator's face is sent to the server side and the image of the object of the client operator on the sending side is changed,
Change the image of the object on the screen in response to a command from the server,
The server providing the virtual reality space is
Virtual reality space sharing method which is characterized in that an instruction on the basis of the data of the transmitted expression, alter the expression of the corresponding object to the operator on the screen of each client. In a virtual reality space sharing system that allows multiple operators to share a virtual reality space via a network and to move each operator's alternate object in the virtual reality space,
A first client and a second client that receive the virtual reality space from a server via the network;
Displaying an alternate object of each operator in the virtual reality space;
Photographing the face of the client operator with a camera;
Detecting a motion vector of a characteristic portion from image data photographed by the camera;
Comparing the obtained motion vector and facial expression determination data to determine the facial expression of the operator of the client and the change in the facial expression;
Only when it is determined that there has been a change in the facial expression, sending facial expression data of the client operator to the server and changing the image of the object of the client operator on the sending side ,
Changing the image of the object on the screen in response to a command from the server ,
To the server that provides the virtual reality space,
Based on the transmitted facial expression data, a computer recording a program for executing a process including a command to change the facial expression of the object corresponding to the operator on the screen of each client is readable. recoding media.

Images