JP3829005B2 - Virtual environment presentation device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
In particular, the present invention is a place of practice in which it is difficult to bring in manuals, such as medical practice (emergency treatment, surgery, massage, nursing care, etc.), dance / gymnastics, etc. The present invention relates to a virtual environment presentation device that is suitable for learning practical skills that can be learned by the body at an educational site or the like, or for supporting maintenance work of facilities that require many drawings.
[0002]
[Prior art]
One-to-one education by instructors is the best way to acquire various performances / manuals or practical skills (hereinafter also referred to simply as performance) that humans perform interpersonally, such as medical practice, because there is no room to refer to the manual in practice. Efficient and essential.
[0003]
That said, not everyone can always get one-on-one education. Therefore, in addition to this one-on-one education, printed materials, videos, etc. are still used as instructional books and manuals, and it is now possible to learn using CD-ROM and recently the Internet WWW (World Wide Web) media. Yes.
[0004]
[Problems to be solved by the invention]
However, it is difficult to bring these manuals into the field of practice, and even if they are brought in, it is difficult to refer to the manual at the same time while learning the performance, and depending on the medium, the following There are various problems.
[0005]
(1) Printed material: Good portability, but cannot be referenced during the performance or turn pages.
[0006]
(2) Video: There is an advantage that you can perform while watching from a distance without any operation, but you can not go back, change speed, change angle, so beginners who want to slow down the tempo or change angle Can't meet the need to see the spot in the blind spot. Also, if the model performance and the user's facing direction are opposite, the procedure must be translated constantly in the head, which may be difficult for beginners.
[0007]
(3) CD-ROM: The video problem has been solved, but mouse operation is required.
[0008]
(4) WWW: Although there is an advantage that the content that can be provided is expanded as compared with the CD-ROM, a mouse operation is similarly required, and portability such as line connection is inferior.
[0009]
The present invention has been made to solve the above-mentioned conventional problems, and provides an environment in which the support can be received by an online manual while continuing acting and working in a place of practice, that is, a walking dictionary environment. This is the issue.
[0010]
[Means for Solving the Problems]
The present invention relates to an information presentation device configured to be capable of presentation control of multimedia information by an image recognition unit and a voice recognition unit in a virtual environment presentation device, a hands-free remote dialogue operation on the device, and the device. A user interface device that can be attached to a user and includes at least a video input unit, a video output unit, a voice input unit, and a voice output unit, which enable see-through viewing of provided information. A user's visual field image is input, and based on the visual field image, the user's line of sight is recognized by the image recognition means of the information presentation device, The image of the angle closest to the recognized line of sight is selected from model images or live-action images in multiple directions prepared in advance. Provided to the video output means And the While controlling the video angle to approach the line of sight, the voice input means inputs the user's voice, and recognizes the character string uttered by the user by the voice recognition means of the information presentation device from the voice, The content of the multimedia information provided by the information presentation device is switched based on the character string, and the video input by the video input means and the video provided by the information presentation device are processed by image processing. By synthesizing and providing the synthesized image to the video output means, the problem is solved.
[0011]
That is, in the present invention, the video input means of the user interface device allows manual information supporting performance etc. to be presented to the user as a video, and the video is sent via the audio output means, the video output means, etc. Thus, for example, the user can switch to the desired information content without using hands, that is, hands-free, based on the voice and line of sight of the user, so that the user supports the performance while continuing the performance, etc. Desired support information can be received.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0013]
FIG. 1 is a block diagram conceptually showing an emergency medical support system (virtual environment presentation device) according to an embodiment of the present invention, and FIG. 2 is a block diagram showing the configuration of the system more specifically. FIG. 3 is a front view showing the headset (user interface device) constituting the support system of the present embodiment together with a state in which the user is wearing, and FIG. 4 is a plan view thereof.
[0014]
The emergency medical support system according to the present embodiment is a home-oriented system that realizes a virtual environment where hands-free learning can be performed while viewing an example video provided as an electronic manual as a manual from a remote location. .
[0015]
As shown in FIGS. 1 and 2, the support system of this embodiment is a personal computer main body (information presentation) configured to be able to present and control multimedia information including manual images for emergency medical assistance. Device, client terminal) A and a headset (user interface device) 10 wirelessly connected to the main body A.
[0016]
In this embodiment, the multimedia information is provided to the computer main body A from the file server (multimedia server) C of the WWW server B via the Internet.
[0017]
In FIG. 1, HTML (Hyper Text Markup Language) is a language used to create a homepage on the Internet, Java is a programming language developed by Sun Microsystems, and a program (applet) created in this language is the Internet. Executed when displaying the homepage of VRML (Virtual Reality Modeling Language) is a language that can express three-dimensional graphics on the Internet, and MIDI (Musical Instruments Digital Interface) is a standard for exchanging music data on a computer.
[0018]
The WWW is currently the most promising media in that it can provide abundant content online, and can provide functions such as backtracking, speed change, and angle change technically.
[0019]
In the present embodiment, the following five functions required as a user interface are realized in order to allow the user to refer to the electronic manual while learning acting or the like.
[0020]
(1) A computer can be operated without using both hands (hands-free operation).
[0021]
(2) It must be able to move to the extent that it does not interfere with performance (portability and lightness).
[0022]
(3) A model performance (virtual video) and a real image can be viewed simultaneously (see-through).
[0023]
(4) The angle of the model performance should be as close as possible to the user's viewpoint (at least not in the opposite direction).
[0024]
(5) The speed of the model performance is variable according to the skill of the user and can be changed during the performance.
[0025]
In the present embodiment, the above-described functions are realized by adopting the following system configuration.
[0026]
That is, as shown in FIGS. 3 and 4, the headset 10 includes a liquid crystal goggles 12, a small video camera 14, headphones 16 composed of left and right ear pads, a hands-free microphone 18 movable in the direction of the arrow, and the liquid crystal goggles. The penlight 20 fixed on the head 12 and the telescopic headband 10A are integrated.
[0027]
The liquid crystal goggles 12 are a see-through type having an optical see-through function using a translucent liquid crystal display. The goggles 12 and the headphones 16 are connected to a receiver (video receiving means, audio receiving means) 22 fixed by a belt around the user's waist via a goggle video cable 12C and a headphone audio cable 16C, respectively. Yes. The receiver 22 has a built-in battery, and supplies power to the goggles 12 via the goggle video cable 12C in addition to supplying power to the video receiving means and audio receiving means. .
[0028]
The small video camera 14 and the microphone 18 are connected to a transmitter (video transmission means, audio transmission means) 24 via a video camera cable 14C and a microphone audio cable 18C, respectively. The transmitter 24 also has a built-in battery. In addition to supplying power to the video transmission means and audio transmission means, the transmitter 24 supplies necessary power to the video camera 14 via the video camera cable 14C. The necessary power is supplied to the microphone 18 via the microphone audio cable 18C.
[0029]
In the present embodiment, the computer body A (client terminal) includes a speech recognition module 30, a status control unit 32, a speech synthesis module (or MIDI sound source) 34, a playback speed control unit 36, as shown in FIG. An image processing unit 38 that performs rendering, video decoding and reproduction, and synthesis with a real background image, and an image recognition module 40 having a camera parameter extraction function are provided.
[0030]
Then, when the user attaches the headset 10 to the head, the user can watch the manual provided on the electronic medium while performing the performance. A is connected to the wireless analog line via the receiver 22 and the transmitter 24.
[0031]
That is, the liquid crystal goggles 12 are input with a video signal transmitted by infrared from the image processing unit 38 via the receiver 22, and the headphone 16 is similarly input with an audio signal from the voice synthesis module 34. It has become. On the other hand, a video signal transmitted by radio waves in the VHF band is output from the video camera 14 to the image processing unit 38 and the image recognition module 40 via the transmitter 24, and audio is similarly transmitted from the microphone 18 to the voice recognition module 30. A signal is output.
[0032]
As described above, in order to avoid interference between the headset 10 and the computer main body A in order to avoid bidirectional interference between video and audio, there are two types of infrared and VHF bands currently available. We are taking measures to share transmission equipment.
[0033]
The computer main unit A is connected to the WWW server B via the Internet / digital line, the performance status is output from the status control unit 32 to the file server C, and the viewpoint information is received from the image recognition module 40. On the contrary, audio information is input from the file server C to the audio synthesizing module 34, and video information is input to the image processing unit 38.
[0034]
In the present embodiment, the function (1) is realized by operating a computer by voice input / recognition. Specifically, a user's voice is input by the microphone 18, a character string uttered by the user is recognized from the voice by the voice recognition module 30 of the computer main body A, and the character string is based on the character string. By sending a switching signal to the file server C by the status control unit 32, the contents of the video / audio multimedia information output from the file server C to the computer main body A and provided to the liquid crystal goggles 12 and the headphones 16 from the main body A Can be switched.
[0035]
In order to realize this function, HTML anchors (characters with underline: a function that jumps to a predetermined page when clicked) are made to jump to a point specified based on a character code obtained by voice recognition. Adopted a method of adding a simple control program to an HTML sentence.
[0036]
FIG. 5 is a block diagram conceptually showing this. When the computer main body A is connected to the server via the Internet, the main management program main body described in the Java applet is started by the main management program starting HTML statement. After that, by inputting a key to the menu button of the main body with a mouse or the like, the multimedia display program main body described in the Java applet can be started by the multimedia display program starting HTML sentence. In this state, information on a desired file can be read from the file server C by performing key input to the menu button of the display program body. As described above, the menu for information transmission can be switched with the WWW server B only by mouse click input, but it can be realized by key input.
[0037]
When a voice command is input from the microphone 18 to the voice recognition module 30, the module 30 converts the voice command into a character code corresponding to a predetermined menu button, and executes an input process equivalent to key input of the character code. Is done. That is, a desired anchor function for jumping to another HTML page by voice comma is realized.
[0038]
In order to realize the function (2), as shown in FIGS. 3 and 4, the headset 10 having the liquid crystal goggles 12, the small video camera 14, the headphones 16, and the microphone 18 is connected to the user's head. The headset 10 and the computer main body A, which is a WWW terminal, are connected wirelessly as described above, and as a result, wireless connection between the headset 10 and the WWW server B is realized. is doing.
[0039]
In order to realize the function (3), the image processing unit 38 synthesizes an actual image input by the video camera 14 and a model image by CG (computer graphics) provided by the computer main body A. The composite image is provided to the liquid crystal goggles 12 with the see-through function closed. By doing so, it is easy to make the background portion completely solid in the CG image. Therefore, if the real image is embedded in the background portion by using a so-called chroma key composition method, the quality of both is improved. Since it can be prevented from deteriorating, a clear model image can be provided. However, although the model image is unclear, the see-through function can be used as it is.
[0040]
In order to realize the function (4), for example, four types of camera work as shown in FIG. 6 are obtained from a silhouette image of a subject (for example, a practitioner) in a line-of-sight image captured by a user with the video camera 14. May be selected automatically.
[0041]
Further, the video camera 14 inputs a user's visual field image, recognizes the user's line of sight by the image recognition module 10 of the computer main body A based on the visual field image, and determines the angle of the video provided to the liquid crystal goggles 12. You may make it control so that it may approach the said eyes | visual_axis.
[0042]
Specifically, only the orientation of the subject's face is recognized from the user's visual field image by the moment method of image processing, and an image having an angle closest to that direction is provided. For this purpose, an exemplary video is produced by 3D CG, and a video that is rendered at an angle in several directions per scene is prepared, or a live-action video taken from several directions is prepared. Although it is difficult to recognize an accurate viewpoint with the current image processing, the 3D CG method has a problem that it takes time to calculate, but if the processing capability of the terminal is large, the graphics engine For example, the angle may be arbitrarily changed by using a method of rendering in real time. In addition, there is a method in which a position sensor is attached to goggles to completely follow the viewpoint, which is not suitable for this application because of a large restriction on the moving range.
[0043]
In the present embodiment, the camera work of the image that the user can see through the liquid crystal goggles 12 is only for the CG image, so that the operation hidden on the back of the practitioner can be seen through. There is also a function for displaying the operator (skeleton) and a function for displaying the patient (patient) in a skeleton so that the operation can be seen anatomically. About this, a specific example is shown later.
[0044]
Similarly, although only in the case of CG images, when multiple parts (heads, hands, etc.) are moving in cooperation, all parts other than the selected part are stopped and played back, There is also a single part motion display function that can be stopped and played back.
[0045]
In addition, there is a function of expanding the time axis with respect to the designated area when reproducing the model video presented on the liquid crystal goggles 12.
[0046]
The playback video of a movie basically displays a temporally changing motion as a continuous still image. Therefore, each still image that changes continuously is stored in the database as image data of frames 1 to 7... As shown in FIG.
[0047]
However, since it only appears once in the model performance, the beginning or end of the repeated performance that is easily overlooked, and the skilled person can be smoothly combined with the repeated performance, easily overlooked, sometimes simplified or omitted. There is a case where slow playback in which the playback speed of the frame in FIG. For such video, for example, as shown in FIG. 2B, the two frames are subdivided so as to be compatible with super slow motion playback that plays back detailed video between frames 2 and 3. The images of the frames 2.1 to 2.7... Prepared separately are prepared.
[0048]
Further, in the present embodiment, in order to realize the function (5), the microphone 18 inputs a user's voice, and the user speaks from the voice by the voice recognition module 30 of the computer main body A. Based on the time interval between the character string and the character string, the reproduction speed control unit 36 recognizes the headphone 16 and the liquid crystal goggles 12 from the voice synthesis module 34 and the image processing unit 38 of the computer main body A. The playback speed of time-series multimedia information such as support video and audio provided to each is controlled. Note that the character string here includes one character such as “A”, “I”, “U”.
[0049]
That is, the frame rate for moving image playback is configured to be variable, and the BGM sound is described in MIDI. For example, when the user utters “1, 2, 3” as a plurality of character strings “1, 2, 3,” at predetermined intervals using the microphone 18, the pronunciation of “1” and “3” starts. It is possible to control the time interval to be measured and to reproduce, for example, 30 frames of moving images and 8 quadrants at that interval.
[0050]
FIG. 8 conceptually shows this function. In this figure, the utterance of “1, 2, 3” is replaced with pulses 1 to 3 of the top user input pulse, and the pulse interval of the system clock changes at the timing of pulse 3. Yes. In synchronization with this, the interval for overwriting ready-made frame data is controlled for moving images, the interval for switching the frame drawn in the background to the front is controlled for CG images, and the interval for playing strong notes is controlled for MIDI. It has become so. Note that in MIDI performance interval control, it is also possible to modulate the sound intensity with a fluctuation component of the amplitude of the pulse.
[0051]
Further, an external synchronization signal can be input to the reproduction speed control unit 36, and a biological signal such as a heartbeat of a user or a person facing the user is input to the speed control unit 36, Similarly, the playback speed of the time-series multimedia information can be controlled based on the biological signal.
[0052]
In addition, as shown in FIG. 1, the support system is also provided with a CRT monitor so that other users can see the video that the user is looking through the goggles.
[0053]
Next, the usage method of the emergency medical assistance system of this embodiment is demonstrated.
[0054]
In recent years, with an aging society, an increase in lifestyle-related diseases (old adult diseases) can no longer be avoided. Among these lifestyle-related diseases, circulatory diseases such as angina pectoris and myocardial infarction have a risk of competing for a moment due to seizures. In such emergency medical care, lifesaving and first-aid treatment until transport to the hospital is very important, and there are no doctors, no imaging equipment, no therapeutic drugs in the surroundings. Appropriate medical practices of people at home will determine the fate of the patient. The support system of the present embodiment exhibits particularly excellent effects under such an environment.
[0055]
First, as shown in FIG. 3, the headset 10 which is a user interface device is attached to the user's head.
[0056]
Next, the user inputs a code to the computer main body A by voice input through the key input or the microphone 18, connects the main body A to the WWW server B, and information such as video and voice regarding the model performance is received from the file server C. Start up ready for input.
[0057]
Then, according to the main management program shown in FIG. 5, the still images of (1) to (10) shown as the outline of the terminal control program in FIG. 9 are transferred to the liquid crystal goggles 12 as a mode menu for emergency medical assistance. It becomes possible to display sequentially.
[0058]
FIG. 10 shows the menu screen of the CRT monitor for convenience. However, the liquid crystal goggles 12 display the video displayed in the window of this screen (here, the video (6) shown in FIG. 9). At the same time, instructions and questions as in the same screen are transmitted to the user through the headphones 16.
[0059]
The user performs the operation according to the instruction and answers the question through the microphone 18 with “Yes” or “No”, thereby sequentially updating the menu images shown in FIG.
[0060]
Here, for convenience, it is assumed that the user has selected the heart massage mode (9) from the menu of FIG. When the user inputs a command indicating “video” shown in the screen of FIG. 10 through the microphone 18, the multimedia display program shown in FIG. 5 is activated, and necessary video and audio information is received from the file server C. Is loaded, and a model image of heart massage is displayed as a moving image on the liquid crystal goggles 12.
[0061]
FIG. 11 shows one frame of this exemplary image, and the frame on which the patient (patient) displayed the skeleton as described above on the same monitor screen as in FIG. Actually, the user selects the operation menu that is displayed in characters on this screen by voice input of the character code, displays the support video in the desired state, and while watching the model performance, You will learn how to massage.
[0062]
Incidentally, as the operation menu, speed, playback, and stop can be selected by voice input, and four images of front, top, side, and enlargement corresponding to the camera work shown in FIG. Can be selected by the angle of the video camera 14 or voice input.
[0063]
In addition, although illustration of a continuous frame image corresponding to a moving image of cardiac massage is omitted, another image display function of the emergency medical support system of the present embodiment will be described.
[0064]
FIG. 12 is one image of a scene in which a practitioner covers a chest and is performing a heart massage on a patient who is lying on his back in the lateral direction. Is difficult to understand. Therefore, as shown in FIG. 13 corresponding to this figure, the practitioner can be displayed as a skeleton.
[0065]
FIG. 14 is an image of the front view of the scene where the practitioner is placing his hand on the patient's chest. This video shows the positional relationship of the practitioner's hand with respect to the patient's heart. Whether it is unknown. Therefore, as shown in FIG. 15, the patient can be displayed in a skeleton manner so that the relationship between the heart inside the ribs and the position of the practitioner's hand can be understood.
[0066]
Such a skeleton display can be dealt with by, for example, creating a database separately in advance in addition to the moving image frames by the four camera works, for example, for an image that is a point for performance.
[0067]
As described above, according to the emergency medical support system of the present embodiment, when the user performs emergency medical treatment or practice thereof, the system can be operated without using both hands, and there is no problem in performance. The model video and the real image can be viewed at the same time, and the video in the direction desired by the user can be viewed, and the speed of the model video can be changed during the performance according to the user's technique. be able to. Therefore, it is possible to easily learn how to perform emergency medical care even at home.
[0068]
Although the present invention has been specifically described above, the present invention is not limited to that shown in the above embodiment, and various modifications can be made without departing from the scope of the invention.
[0069]
For example, although the case where the virtual environment display device is an emergency medical support system has been described in the embodiment, the present invention is not limited to this. Further, the specific external shape and configuration of the headset are not limited as well.
[0070]
In the above embodiment, the WWW server is used as the multi-information server. However, another network may be used, or the database may be directly communicated without using the network. .
[0071]
Moreover, although the case where the computer main body (terminal) is a stationary type has been shown, since the miniaturization of the computer is progressing, the computer main body itself is worn and the main body and the data server are connected wirelessly. It may be.
[0072]
【The invention's effect】
As described above, according to the present invention, it is possible to provide an environment in which the support can be received by video and audio while continuing acting and working in a place of practice.
[0073]
In addition, according to the present invention, when a forgotten matter or a document to be referred to comes out in a practice place where there is no instructor, it is possible to search and refer to the document without interrupting the work. If it is operated in the Internet environment, it can provide the possibility of obtaining a solution even in difficult situations that cannot be handled by a single instructor.
[0074]
In addition, when it is necessary to refer to many drawings in a dim and narrow work site, conventionally, it was necessary to proceed with the work while wirelessly communicating with the person in charge of the design office, but according to the present invention, Since work can be performed while virtually viewing the drawing at the site, the work can be carried out efficiently.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a schematic configuration of an emergency medical support system according to an embodiment of the present invention.
FIG. 2 is a block diagram conceptually showing a specific configuration of the emergency medical support system.
FIG. 3 is a front view showing a headset constituting the emergency medical support system together with a wearing state.
FIG. 4 is a plan view of the headset as seen from above.
FIG. 5 is a block diagram conceptually showing a method for switching WWW content by sound.
FIG. 6 is an explanatory diagram showing the types of camera work of playback video
FIG. 7 is an explanatory diagram showing the relationship between a standard and a playback video sequence with an expanded time axis
FIG. 8 is an explanatory diagram for explaining data reproduction speed control;
FIG. 9 is an explanatory diagram showing an outline of a control program for emergency medical assistance.
FIG. 10 is an explanatory diagram showing a monitor screen corresponding to a virtual image.
FIG. 11 is an explanatory diagram showing a monitor screen corresponding to another virtual image
FIG. 12 is an explanatory diagram showing one scene of a model video
13 is an explanatory diagram showing an example of the skeleton display of FIG.
FIG. 14 is an explanatory diagram showing another scene of the model video
15 is an explanatory diagram showing an example of the skeleton display in FIG. 14;
[Explanation of symbols]
A ... Computer body
B ... WWW server
C ... File server
10 ... Headset
12 ... Liquid crystal goggles
14 ... Small video camera
16. Headphone
18 ... Microphone
20 ... Penlight
22 ... Receiver
24 ... Transmitter
30 ... Voice recognition module
32 ... Status control unit
34 ... Speech synthesis module
36. Playback speed control unit
38. Image processing unit
40. Image recognition module

Claims (8)

Information presentation device configured to be able to control presentation of multimedia information by image recognition means and voice recognition means, remote interactive operation by hands-free to the device, and see-through viewing of information provided by the device A user interface device that has at least a video input means, a video output means, a voice input means, and a voice output means, and that can be attached to a user,
The video input means inputs a visual field image of the user, recognizes the user's line of sight by the image recognition means of the information presentation device based on the visual field image, and prepares a model video or a live-action video in a plurality of directions prepared in advance. among, and provide an image of the closest angle to the recognized visual line to the video output unit, controls so as to approach the angle of the image in the visual axis,
The voice input means inputs a user's voice, recognizes a character string uttered by the user by the voice recognition means of the information presentation apparatus from the voice, and provides the multi-value provided by the information presentation apparatus based on the character string Switch the content of the media information,
In addition, the video input by the video input unit and the video provided by the information presentation device are combined by image processing, and the combined image is provided to the video output unit. Environmental presentation device. In claim 1,
The voice input unit inputs a user's voice, recognizes a plurality of character strings uttered by the user from the voice by the voice recognition unit of the information presentation device, and based on a time interval between the character string and the character string A virtual environment presentation device characterized in that the playback speed of time-series multimedia information provided by the information presentation device is controlled. In claim 1,
A biological signal including a heartbeat of a user or a person facing the user is input, and the reproduction speed of time-series multimedia information provided by the information presentation device is controlled based on the biological signal. A virtual environment presentation device characterized. In claim 1,
A virtual environment presentation device, wherein the user interface device including the video input unit, the video output unit, the voice input unit, and the voice output unit includes a headset that can be attached to a user's head. In claim 4,
A virtual environment presentation device, wherein the headset is capable of wirelessly transmitting information to and from the information presentation device via a video transmission unit, a video reception unit, a voice transmission unit, and a voice reception unit. In claim 4,
The virtual environment presentation device, wherein the information presentation device is connected to a data server that provides multimedia information. In claim 6,
The virtual environment presentation device, wherein the headset is wire-connected to the information presentation device attached to a user's body, and the information presentation device is wirelessly connected to the data server. In claim 6,
The virtual environment presentation device, wherein the information presentation device and the data server are a WWW client and a WWW server connected to the Internet, respectively.

Images