JP3310325B2 - Head mounted display device and system using the same - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a head mounted display (hereinafter referred to as "HMD") device and a device using the same.
It is about the system that was.

[0002]

2. Description of the Related Art Image display means is fixed to a user's head.
An HMD device that allows the user to view a display image in the entire visual field has already been proposed. One of them is an image in a visual field image of the user's external world (hereinafter, referred to as an “external world image”). There is a so-called see-through type HMD device in which display images (hereinafter, referred to as “electronic images”) of a display unit are superimposed so that these images can be viewed simultaneously.

[0003]

SUMMARY OF THE INVENTION Conventional see-through type H
With the MD device, it was not possible to capture an external image viewed by the user. Further, it was not possible to take an image of a superimposed image of the outside world and an electronic image, which is viewed by the user. Therefore, there is a disadvantage that the image information of the electronic image cannot be changed corresponding to the external image.

[0004] The present invention has been made in view of such problems of the prior art, it is an object of the see-through HMD device, the same image as the image the user that looked, That is, an object of the present invention is to provide an apparatus capable of photographing an image in which an external image and an electronic image are superimposed.

[0005]

In order to achieve the above object, a head-mounted display device according to the present invention comprises an image display means for displaying a desired image, and light from the image display means to the eyes of a user. an observation optical system that guides, comprising imaging means for imaging the light from the outside towards the eyes of the user, and a housing for fixing the head of the user accommodates a respective unit, the observation optical system In the light path, the user
Light from the outside world heading for the eyes and the image display means
An optical path combining means for superimposing light on the eye of the user;
Light from the outside world and light from the image display means
An optical path toward the user's eye in a superimposed state;
So that the light is guided separately to the optical path toward the imaging means.
And an optical path separating means .
Further, in the head mounted display device according to the present invention, the optical path separating unit is overlapped by the optical path combining unit.
The combined light path to the user's eye and the light path
It is preferable to be configured so as to be guided separately to an optical path toward the image means . Further, in the head-mounted display device according to the present invention, the optical path separating means may include
The outside world, which also has road combining means, and is directed to the user's eyes
An optical path for directing light from the
The light path leading to the step is separated and guided, and the image
The light path from the display means to the eyes of the user and the light path
The light path to the image pickup means is separated and guided, and
Said user's eye being guided in a light path towards the user's eye
Light from the outside world heading to the eyes of the user
The light from the image display means guided to the optical path is superimposed.
And guided to the optical path toward the imaging means.
Light from the outside facing the user's eye and the imaging means
With light from the image display means guided to the optical path toward
Preferably that superimposes. In the present invention,
The head mounted display device is the image display means.
The optical axis of the light coincides with the optical axis of the light received by the imaging means.
It is preferred that each means is arranged to
No. Also, a head mounted display according to the present invention
Device, the state of the outside world image captured Ri by said image pickup means
A control unit for controlling an image to be displayed on the image display means.
It is preferable to have control means. In the present invention,
In the head mounted display device, the control means may
The average luminance of the external world image is calculated and the luminance of the electronic image to be displayed
Is determined by the image display means by the control means.
Generate and display an electronic image based on specified brightness information
Is preferred. Also, the head mount disk according to the present invention is used.
In the spray device, the control unit may be configured to control the
From the captured external image, the size of a certain object or the object
The image display means calculates the distance to the control means.
It is preferable to display the information calculated by the steps as an electronic image.
Good. A head-mounted display system according to the present invention is a head-mounted display system using the head-mounted display device according to the present invention. The head-mounted display system includes an image input unit for inputting image information including characters, and an image input unit for inputting image information including characters. Depending on the information provided,
Control means for controlling an image to be displayed on the image display means. In the present invention,
The head-mounted display system is adapted for inputting the image.
Means for converting the input handwritten character information into character information.
And the control means converts the image data by the image input means.
The image display means as an electronic image using the extracted character information
It is preferable to control so as to be displayed on the screen. Further, a head-mounted display system according to the present invention is a head-mounted display system using the head-mounted display device of the present invention, wherein a second image display means provided separately from the housing, And control means for controlling display of the image information of the external world image picked up by the second image display means as an electronic image. In the present invention,
Head-mounted display system, the second
An image display means including a head assembly including the housing;
Use for third parties who do not have a display device
Provide the outside world image seen by the third party
Update the input means for inputting information to be provided to the user.
The control means is input by the input means.
It is displayed on the first image display hand stages information as an electronic image
It is preferable to perform control in such a manner. Further, the head mounted display device according to the present invention is a transmission device for transmitting image information of an external world image captured by the imaging device to another head mounted display device, and an external world image transmitted from another head mounted display device. It is preferable to have a receiving unit for receiving the image information, and an image generating unit for generating the image in order to display an image corresponding to the image information received by the receiving unit on the image display unit.

[0006]

The light from the outside world and the light from the image display means are provided by an optical path combining means and an optical path dividing means arranged in the observation optical path.
Are superimposed and guided to the observer and the imaging means.
The user can separate the electronic image and the external image
Can be viewed, and the same image as the image obtained by superimposing the electronic image and the external image viewed by the user can be taken. Therefore, the external image and the electronic image can be simultaneously recorded, and the image information of the electronic image can be generated corresponding to the external image and provided to the user.

[0007]

An embodiment will be described with reference to the drawings. FIG. 1 shows the appearance of the HMD device of the present invention. In FIG. 1, 1 is an HMD device, 2 is a housing in which optical elements and electric elements of an optical system to be described later are stored, 3 is a protective permeable membrane disposed on an outer peripheral surface and an inner peripheral surface of the housing 2, respectively.
Is a speaker for providing sound information to the user, 5 is a band for fixing the housing 2 to the user's head, 6 is a signal for inputting image information and sound information from an image / sound generating device (not shown). Is a cable for outputting a photographed image to an image processing device (not shown) as a signal. The transmission and reception of signals with these external devices may be performed wirelessly.

FIG. 2 shows the present invention housed in a housing.
4 shows an optical system according to the first embodiment. In the figure, 11 is a user's head, 12 is an eye, 13 is an ear, 14R and 14L are two-dimensional liquid crystal display elements arranged on the left and right corresponding to both eyes, 15
Is a light source for the backlight of the liquid crystal display elements 14R and 14L,
16 is a condenser lens, 17 is a field lens, 18 is a concave half mirror, 19 is a half mirror for imaging, 20, 2
1 is a crystalline lens and a retina of the eye 12, respectively, 22 is light reflected by the imaging half mirror 19, and liquid crystal display elements 14R, 14L.
23R and 23L are CCD image sensors for photographing images arranged on the left and right corresponding to both eyes, and 37 is a collimator lens for converting the light of the backlight light source 15 into a parallel light flux. . Image information input to the housing 2 is displayed on the liquid crystal display elements 14R and 14L, respectively, and sound information is output via the speaker 4. The images captured by the image sensors 23R and 23L are output as signals to an external device via the cable 7.

In FIG. 2, the light emitted from the backlight light source 15 is converted into a substantially parallel light beam by the collimator lens 37 to illuminate the liquid crystal display elements 14R and 14L from behind. The light transmitted through the liquid crystal display elements 14R and 14L is condensed by a condensing lens 16 near a decentered field lens 17 to form an image of a light source 15. Further, this light is deflected by the field lens 17 and reflected by the concave half mirror 18, and a part of the light passes through the imaging half mirror 19 to form a secondary image of the light source 15 near the pupil of the eye 12. The light reflected by the imaging half mirror 19 is transmitted through the imaging lens 22 to the image sensor 23.
R, 23L. In the figure, the optical path of the light emitted from the light source 15 is shown by a solid line.

Light from image information displayed on the liquid crystal display elements 14R and 14L is converted into a substantially parallel light beam through a condenser lens 16, a field lens 17 and a concave half mirror 18, and a part of this light beam is converted. Imaging half mirror 1
9 through the lens 20 of the eye 12 to form an image on the retina 21 to form an electronic image. Also, the imaging half mirror 1
The light reflected by 9 forms an image on the image sensors 23R and 23L via the imaging lens 12. On the other hand, a part of the light from the outside entering the concave half mirror 18 is transmitted through the half mirror 18 and the liquid crystal display elements 14R, 14R.
L overlaps with light from the eye 12 and the image sensor 23
R and 23L, respectively, to form an external image. In the drawing, the optical path of light from the outside is shown by broken lines.

Accordingly, the user can view an image in which the electronic image and the external image are superimposed, and can photograph the same image as the user is viewing with the image sensors 23R and 23L. Further, since the image of the light source 15 is formed near the pupil of the eye 12, the light from the light source 15 can be used without waste, and a bright image can be obtained.

In this case, the imaging half mirror 19 is used.
May be rotatably installed like a flip-up mirror of a single-lens reflex camera, so that it can be inserted into and removed from the optical path. For example, as shown in FIG. 3, when imaging is not performed, the imaging half mirror 19 is rotated around the fulcrum 40 to be disposed outside the optical path, and all light reflected by the concave half mirror 18 is transmitted to the eye 12. In the case of guiding (FIG. (A)), when taking a picture, if a part of the light reflected by the concave half mirror 18 is guided to the image sensor 23R by inserting it into the optical path (FIG. (B)), the photographing is performed. In this case, the amount of light incident on the eye 12 can be increased, and the user can obtain a brighter image.

FIG. 4 shows an optical system according to a second embodiment of the present invention . In this embodiment, the image sensor 23R is arranged outside the liquid crystal display element 14R. In FIG. 4, light from the liquid crystal display element 14R passes through the condenser lens 16 and the field lens 17 and partially reflects the concave half mirror 18 to form an image on the retina 21 of the eye 12, and partially reflects the concave surface. The light passes through the half mirror 18. The light transmitted through the concave half mirror 18 is reflected by a curved portion R on the nose side of the concave imaging half mirror 19 which is formed in a concave shape, and then deflected by the eccentrically arranged field lens 17 to form an imaging lens. The light enters the image sensor 23 </ b> R via 22. On the other hand, light from the outside passes through the imaging half mirror 19, then partially transmits through the concave half mirror 18, and partially reflects through the concave half mirror 18, and the light from the liquid crystal display element 14R, respectively. Overlap with each other and enter the eye 12 and the image sensor 23R to form an external image.

FIG. 5 shows an optical system according to a third embodiment as a reference example . In the present embodiment, only an external image is captured. In FIG. 5, a part of light from the external environment is reflected by the imaging half mirror 19 and guided to the image sensor 23R.
Part of the light passes through the imaging half mirror 19 and overlaps with light from the liquid crystal display element 14, enters the eye 12, and forms an image on the retina 21.

FIG. 6 shows an optical system according to a fourth embodiment as a reference example . In this embodiment, an external image and an electronic image are selected, and only one of the images can be captured by the image sensor 23R. In the figure, reference numerals 41, 42 and 43 denote polarizing plates.
3, and the polarizing plate 42 is rotatably arranged about the optical axis.

In FIG. 6, light from the liquid crystal display element 14R enters a polarizing plate 41, and light passing through the polarizing plate 41 passes through a condenser lens 16, a field lens 17, and a concave half mirror 18 to form an image pickup half. The light enters the mirror 19. The light transmitted through the imaging half mirror 19 forms an electronic image by forming an image on the retina 21 of the eye 12, and the light reflected by the imaging half mirror 19 is formed by an imaging lens 22.
Through the polarizing plate 42. On the other hand, light from the outside enters the polarizing plate 42, becomes polarized light having a vibration direction different from that of the polarizing plate 41 by 90 °, enters the concave half mirror 18, and a part of the light transmitted through the concave half mirror 18 is The light passes through the imaging half mirror 19, overlaps with the light from the liquid crystal display element 14 </ b> R, enters the eye 12, and is partially reflected by the imaging half mirror 19 and guided to the polarizing plate 42.

Here, for example, when only an electronic image is to be photographed, the user rotates the polarizing plate 42 so that the vibration direction of the polarizing plate 42 matches the polarizing plate 41. Then, of the light guided to the polarizing plate 42, the light from the outside world does not pass through the polarizing plate 42 because the vibration direction is different from that of the polarizing plate 42, while all the light from the liquid crystal display element 14R passes through the polarizing plate 42. Since the light passes through, the light enters the image sensor 23R, and only an electronic image is formed on the image sensor 23R. Also,
When it is desired to take only the external image, the polarizing plate 42 may be rotated so that the vibration direction thereof coincides with that of the polarizing plate 43. Further, when an image in which both images are superimposed is taken, the polarizing plate 42 is used. May be arranged at a position inclined by 45 ° with respect to the vibration directions of the polarizing plates 41 and 43. In the second to fourth embodiments, the same configuration is applied to the optical system of the left eye of the user (not shown).

Next, a system configuration of the HMD device and the external device will be described. FIG. 7 shows a fifth embodiment of the present invention, which is a system configuration using the HMD device including the optical systems of the first and second embodiments, and a third embodiment which is a reference example.
And a system using an HMD device comprising the optical system of the fourth embodiment.
3 shows a stem configuration . In FIG. 7, an image generating device 29 converts information provided to a user into an image signal,
Transmit to the D device 1. The transmitted image information is converted into visible light by the liquid crystal display elements 14R and 14L, and overlaps with external light and enters the user's eyes and the image sensors 23R and 23L. The external image and the electronic image captured by the image sensors 23R and 23L are converted into electric signals, and then transmitted to the storage device 30 to be recorded and stored as a recording medium such as a magnetic tape, a film such as a hard copy, or a print output. You.

FIG. 8 shows a sixth embodiment of the present invention.
In FIG. 8, an analysis computer 31 analyzes information obtained from the image signals transmitted from the image sensors 23R and 23L to generate an information signal corresponding to the state of the external image, and generates an image. The data is transmitted to the device 29. For example, if the average luminance of the external image is calculated to determine the luminance of the electronic image to be displayed, and the image generation device 29 generates and displays the electronic image based on the luminance information,
The user can obtain an easy-to-see brightness electronic image according to the external world image. In addition, the three-dimensional size of a certain object and the distance to the object are calculated from the external images captured by the image sensors 23R and 23L, and this information is displayed as an electronic image and provided to the user. be able to. The image input device 36 is a device for a user or a third party to input characters using a pen or the like. The input handwritten character information is converted into a character signal and transmitted to the analysis computer 31. The data is provided to the user via the image generation device 19.

FIG. 9 shows a seventh embodiment of the present invention.
The present embodiment is an embodiment in which a user A of the HMD device 1 and a user B of the HMD device 1 'can confirm the external image seen by each other. In this case, the external world image seen by the other party is formed as, for example, a reduced image in each electronic image, and can be viewed as a part of the provided information. In FIG. 9, an external image viewed by the user A is captured by the image sensors 23R and 23L,
The data is transmitted from the D device 1 to the transmission device 33. This image signal is transmitted to the receiving device 34 'after undergoing predetermined signal processing in the transmitting device 33. The image signal transmitted to the receiving device 34 'is sent to the image generating device 29', and is captured as a reduced image in the electronic image as a part of the information provided to the user B. Then, the generated electronic image is transmitted to the HMD device 1 'and provided to the user B via the liquid crystal display elements 14R' and 14L '. The external image viewed by the user B is captured into the electronic image viewed by the user A via the image sensors 23R 'and 23L', the transmitting device 33 ', the receiving device 34, and the image generating device 29. It has become. The embodiment of the present invention can be applied to users of two or more HMD devices. In this case, for example, by providing a reception signal selection function to the reception devices 34 and 34 ', a desired partner can be provided. Of the outside world can be confirmed.

FIG. 10 shows an eighth embodiment of the present invention. In this embodiment, a third party who is not equipped with the HMD device,
This is a mode in which an external image viewed by a user is confirmed, and information corresponding to the external image can be provided to the user at any time. FIG.
In, the external image captured by the image sensors 23R and 23L is transmitted to the computer 39, converted into a video signal by the computer 39, and output to the image display device. The third party can obtain the same information as the information obtained by the user viewing the external image from the image display device 35, and can know the environment where the user is located. Then, the third party appropriately inputs information to be provided to the user from the external input device 36 such as a keyboard, and displays the information in an electronic image viewed by the user. Information can be obtained. In the fifth to eighth embodiments, transmission and reception of signals are performed by wire or wirelessly, and sound information (not shown) is also transmitted to the HMD device 1 together with the image information and provided to the user. ing.

As described above, according to the present invention, light from the outside world and light from the image display means are superimposed and observed.
I separated it so that it led to the observer and the imaging means
Then, the user creates an image in which the electronic image and the external image are superimposed.
It can be seen and the electronic image that the user is looking at
It is possible to shoot the same image as the image superimposed on the world image
Wear. Therefore, the external image and the electronic image can be simultaneously recorded, and the image information of the electronic image can be generated corresponding to the external image and provided to the user.

[Brief description of the drawings]

FIG. 1 is a diagram showing an appearance of an HMD device of the present invention.

FIG. 2 is a diagram showing an optical system of a first embodiment of the HMD device of the present invention.

3A and 3B show a case in which the imaging half mirror of the first embodiment is rotatably installed, wherein FIG. 3A shows a case where the imaging half mirror is arranged outside the optical path, and FIG. The main optical systems at the time are shown.

FIG. 4 is a diagram showing an optical system of a second embodiment of the HMD device of the present invention.

FIG. 5 is a diagram showing an optical system of a third embodiment of the HMD device as a reference example .

FIG. 6 is a diagram showing an optical system of a fourth embodiment of the HMD device as a reference example .

FIG. 7 is a diagram showing a system configuration of a fifth embodiment of the HMD device of the present invention , and furthermore, a system configuration of an HMD device as a reference example .

FIG. 8 is a diagram showing a system configuration of a sixth embodiment of the HMD device of the present invention.

FIG. 9 is a diagram showing a system configuration of a seventh embodiment of the HMD device of the present invention.

FIG. 10 is a diagram showing a system configuration of an eighth embodiment of the HMD device of the present invention.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... HMD apparatus 2 ... Housing 14R, 14L ... Two-dimensional liquid crystal display element 18 ...
Concave half mirror 19 ... Half mirror for imaging 23R, 23L ...
CCD image sensor 29 image generating device 30 recording device 31 analysis computer 33 transmitting device 34 receiving device 35 image display device 36 image input device 39 ... Computers 41, 42, 43 ... Polarizing plates

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP 3-64171 (JP, A) JP 3-188777 (JP, A) JP 3-226198 (JP, A) JP 3-A 235491 (JP, A) JP-A-4-22358 (JP, A) JP-A-4-26289 (JP, A) JP-A-5-46161 (JP, A) JP-A-59-107686 (JP, A) Japanese Utility Model Application Hei 2-26870 (JP, U) Japanese Utility Model Application Showa 61-155722 (JP, U) (58) Field surveyed (Int. Cl. ⁷ , DB name) H04N 5/64 511

Claims (12)

(57) [Claims] 1. An image display device for displaying a desired image, an observation optical system for guiding light from the image display device to a user's eye, and an imaging device for capturing light from the outside toward the user's eye Means, and a housing for accommodating the respective means and fixing the same to the user's head , wherein light from the outside toward the user's eyes and the image display are provided in the optical path of the observation optical system.
Light path combining means for superimposing light from the means, light from the outside toward the eyes of the user, and the image display
Eye of the user in a state where light from the means is superimposed
Separated into an optical path toward
And a light path separating means arranged to be guided . 2. The optical path separating means according to claim 1, wherein said optical path separating means is an optical path combining means.
Light directed toward the user's eye by the light superimposed by
Path and an optical path toward the image pickup means.
The head-mounted display device according to claim 1, wherein the head-mounted display device is configured as follows. 3. The optical path separating unit according to claim 2, wherein the optical path separating unit is an optical path combining unit.
And light from the outside facing the user's eyes.
An optical path towards the user's eye and towards the imaging means
And the light path is separated from the light path.
An optical path for directing the light toward the eyes of the user and the imaging means
To the light path heading toward
Heading to the user's eye guided to the light path to the eye
Guided by light from the outside world and light path towards the user's eyes
And light from the image display means to be superimposed.
Wherein the user is guided to an optical path toward the imaging means.
Light from the outside world heading to the eye and heading to the imaging means
The light from the image display means guided to the optical path is superimposed.
Head-mounted display device according to claim 1, characterized in Rukoto allowed. 4. The apparatus according to claim 1, wherein each of the means is arranged such that an optical axis of light from the image display means coincides with an optical axis of light received by the imaging means. Head mounted display device. 5. The head-mounted display device according to claim 1, further comprising control means for controlling an image to be displayed on the image display means according to a state of an external image picked up by the image pickup means. . 6. The control means calculates the average brightness of the external image and determines the brightness of the electronic image to be displayed, and the image display means controls the electronic image based on the brightness information determined by the control means. 6. The head-mounted display device according to claim 5, wherein the head-mounted display device generates and displays the data. 7. The control unit calculates a size of a certain object or a distance to the object from an external image captured by the imaging unit, and the image display unit calculates the distance by the control unit. The head mounted display device according to claim 5, wherein the information is displayed as an electronic image. 8. A head-mounted display system using the head-mounted display device according to claim 1, wherein an image input unit for inputting image information including characters, and information input by the image input unit. Control means for controlling an image to be displayed on the image display means in accordance with the following. 9. The image input means converts input handwritten character information into character information, and the control means displays the character information converted by the image input means on the image display means as an electronic image. The head-mounted display system according to claim 8, wherein the control is performed as follows. 10. A head-mounted display system using the head-mounted display device according to claim 1, wherein a second image display unit provided separately from the housing, and an image taken by the imaging unit Control means for controlling the image information of the external world image to be displayed as an electronic image on the second image display means. 11. The second image display means is for providing an outside world image seen by a user to a third party who is not equipped with a head mounted display device including the housing, Input means for inputting information to be provided to the user by the three persons, wherein the control means controls the information input by the input means to be displayed on the first image display means as an electronic image. The head-mounted display system according to claim 10, wherein: 12. A transmitting means for transmitting image information of an external image captured by the image capturing means to another head-mounted display device, and receiving image information of an external image transmitted from another head-mounted display device. The image processing apparatus according to claim 1, further comprising: a receiving unit; and an image generating unit that generates the image corresponding to the image information received by the receiving unit, so as to display the image on the image display unit. Head mounted display device.