JP7753000B2 - Video display device and control method thereof - Google Patents

Description

The present invention relates to a technology for presenting images by directing different images to the user's left and right eyes.

Video display devices with a user's gaze detection function can adjust the diopter by calculating the diopter adjustment amount corresponding to the parallax of the gaze point position. In Patent Document 1, gaze detection is performed via a display optical system that uses a prism. A special prism is used to ensure that changes in optical power caused by diopter adjustment do not affect gaze detection. In addition, because a configuration is used that adjusts the diopter according to the user's gaze point, the sense of discomfort felt during stereoscopic viewing is reduced, and motion sickness during viewing can be suppressed.

Japanese Patent Application Publication No. 8-234141

In conventional technology, the use of prisms results in large and heavy optical systems. When a user wears a video display device on their head, such as a head-mounted display, it is difficult to wear a device with a large optical system due to its weight. Even if it were possible to wear it, it would cause significant fatigue for the user, making it difficult to view for medium to long periods of time.
An object of the present invention is to provide a smaller and lighter image display device that allows for diopter adjustment according to the position of the gaze point.

An image display device according to an embodiment of the present invention includes first and second image display units that display first and second images to the right and left eyes of a user, respectively; first and second display optical systems that correspond to the first and second image display units; first and second line-of-sight detection means that perform line-of-sight detection for the first and second image display units, respectively, without using the first and second display optical systems; and a parallax detection means that detects a parallax between the first image and the second image at a position of a fixation point obtained from the detection results of the first and second line-of-sight detection means. a calculation unit for calculating a diopter adjustment amount to be adjusted by the calculation unit, first and second diopter adjustment units for adjusting the diopter of the first and second image display units in accordance with the diopter adjustment amount, and first and second optical path changing units arranged on the user side of the first and second display optical systems, respectively; wherein the first and second line-of-sight detection units each include an infrared illumination unit and a camera, and the camera of the first line-of-sight detection unit and the camera of the second line-of-sight detection unit are arranged in a direction in which the first and second image display units are arranged side by side, the optical axis of a camera provided in the first display optical system is tilted at a predetermined angle with respect to the optical axis of the first display optical system , and the optical axis of a camera provided in the second display optical system is tilted at a predetermined angle with respect to the optical axis of the second display optical system; the first gaze detection means performs gaze detection via the first optical path changing means; the second gaze detection means performs gaze detection via the second optical path changing means; the first and second display optical systems are composed of flat optical elements whose focal length can be changed by an electric signal; the first and second display optical systems also serve as the first and second diopter adjustment means, respectively; the optical elements constituting the first display optical system are arranged in a region between the first optical path changing means and the first image display unit in the direction of the optical axis of the first display optical system; and the optical elements constituting the second display optical system are arranged in a region between the second optical path changing means and the second image display unit in the direction of the optical axis of the second display optical system .

This invention makes it possible to provide a smaller, lighter image display device that allows for diopter adjustment according to the position of the gaze point.

1 is a schematic diagram showing the appearance of a video display device according to an embodiment of the present invention; 1 is a cross-sectional view of an image display device according to a first embodiment, as viewed from above. 1A and 1B are schematic diagrams illustrating the convergence angle of the eyeballs and the diopter during gaze. 1 is a block diagram showing the overall system configuration of a video display device. 10 is a flowchart illustrating operations from line-of-sight detection to diopter adjustment. FIG. 10 is a cross-sectional view of an image display device according to a second embodiment, as viewed from above. FIG. 11 is a cross-sectional view of an image display device according to a third embodiment, as viewed from above.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In this embodiment, a head-mounted display is shown as an example of an image display device in which a pair of images with parallax are displayed on multiple image display units, respectively, and stereoscopic display is possible via multiple display optical systems.

[First Example]
1 is a schematic diagram showing the appearance of an image display device according to this embodiment. The image display device 101 has a belt part 101a for being worn on a user's head 102. The user observes images displayed by the image display device 101 with their right eye 103a and left eye 103b.

The image display device according to this embodiment will be described with reference to Figure 2. Figure 2 is a schematic cross-sectional view of the image display device 101 as viewed from above. In Figure 2, the reference numerals of components corresponding to the user's right eye 103a are marked with an a, and the reference numerals of components corresponding to the user's left eye 103b are marked with a b.

The image display unit consists of an image display unit 201a for the right eye and an image display unit 201b for the left eye. The image display unit is composed of a liquid crystal display, an organic EL (Electro Luminescence) display, or the like. The display optical system 202a is an optical system that guides the image on the image display unit 201a to the right eye 103a, and its optical axis is denoted as La. The display optical system 202b is an optical system that guides the image on the image display unit 201b to the left eye 103b, and its optical axis is denoted as Lb. For simplicity, both display optical systems 202a and 202b in Figure 2 are shown as single lenses, but there are various optical systems, such as optical systems with multiple lenses and optical systems that are thinned using Fresnel elements.

Gaze detection camera 203a detects the gaze of right eye 103a, and its optical axis is denoted as LGa. Gaze detection camera 203b detects the gaze of left eye 103b, and its optical axis is denoted as LGb. Gaze detection cameras 203a and 203b are positioned near the sides of display optical systems 202a and 202b, respectively, and can capture the user's gaze without going through display optical systems 202a and 202b. The optical axis LGa of gaze detection camera 203a is tilted at an angle θ1a with respect to the optical axis La of display optical system 202a, which displays an image directly in front of right eye 103a. Similarly, the optical axis LGb of gaze detection camera 203b is tilted at an angle θ1b with respect to the optical axis Lb of display optical system 202b, which displays an image directly in front of left eye 103b. In this way, gaze detection cameras 203a and 203b detect the gaze without going through display optical systems 202a and 202b. Therefore, unlike conventional configurations, the display optical systems 202a and 202b have a simpler configuration, making it possible to realize a lighter and more compact display optical system.

The infrared illumination unit 204a illuminates the gaze detection camera 203a in the right eye 103a, forming a corneal reflection image. Similarly, the infrared illumination unit 204b illuminates the gaze detection camera 203b in the left eye 103b, forming a corneal reflection image. Infrared LEDs (light-emitting diodes) or the like are used for the infrared illumination units 204a and 204b. Although only one infrared illumination unit 204a and one infrared illumination unit 204b are shown in Figure 2, in reality, multiple units are arranged surrounding the display optical systems 202a and 202b, respectively.

In this way, the gaze detection cameras and infrared illumination units can detect the gaze of the user's right and left eyes. In other words, the gaze detection cameras 203a and 203b and the infrared illumination units 204a and 204b constitute gaze detection means.

The gaze detection method uses a well-known method that uses an image of the eye's pupil and an image of the corneal reflection created by infrared illumination, so a detailed explanation of this method will be omitted. Furthermore, the gaze detection means of this embodiment captures an image of the eyeball from a position tilted relative to the user's eye. Therefore, perspective distortion occurs in the pupil and corneal reflection images captured by the gaze detection camera, but by applying correction, the gaze can be detected accurately.

The diopter adjustment unit 205a adjusts the diopter by driving the image display unit 201a in the direction of the optical axis La. The diopter adjustment unit 205b adjusts the diopter by driving the image display unit 201b in the direction of the optical axis Lb. Diopter adjustment involves adjusting the power to match the visual acuity of the user's left and right eyes. The diopter adjustment units 205a and 205b have motors capable of linear drive. There are no limitations on the motor drive method, but since the motors are located near the user's ears when the image display device 101 is in use, a quiet motor is preferred; for example, an ultrasonic motor that uses friction or a voice coil motor that uses electromagnetic force may be used.

The diopter adjustment units 205a and 205b perform diopter adjustment based on the diopter adjustment amount at the position of the gaze point determined from the detection results by the line of sight detection means. The diopter is adjusted by changing the distance between the image display units 201a and 201b and the display optical systems 202a and 202b. Diopter adjustment is performed in real time each time the gaze point is changed, thereby reducing the user's dizziness and fatigue when watching videos. Details of this are explained using Figure 3.

Figure 3 is a schematic diagram illustrating the convergence angle and diopter of the eyes when gazing. Figure 3(A) shows the state of gaze in the real world, and Figure 3(B) shows the state of gaze on the image display device 101. In Figure 3(A), the observer is gazing at object point 301, which is located a distance A1 away, in the real world. The line of sight Ga1 of the observer's right eye 103a and the line of sight Gb1 of the observer's left eye intersect at object point 301, forming an angle B1. Angle B1 corresponds to the convergence angle when the observer gazes at object point 301. At this time, the crystalline lens changes to focus at distance A1 with respect to the diopter of the observer's eyeball, and as shown by rays Da1 and Db1 in the figure, light rays emitted from object point 301 converge at approximately one point on the retina inside the eyeball. In other words, in the real world, the distance corresponding to convergence angle B1 and distance A1 corresponding to diopter always coincide.

On the other hand, in Figure 3(B), the observer is gazing at object point 301 located at a distance A1 away on the image display device 101. The convergence angle B1 is the same as in the real world case shown in Figure 3(A). Virtual image position 302 is the virtual image position when the observer observes the first image display unit 201a and the second image display unit 201b through the first display optical system 202a and the second display optical system 202b, respectively. Virtual image position 302 is located at a distance A2 (> distance A1) from the eyes 103a and 103b. With regard to the diopter of the eyeball, the crystalline lens changes so that the focus is at distance A2, and as shown by light rays Da2 and Db2 in the figure, the light rays emitted from virtual image position 302 converge at approximately one point on the retina inside the eyeball. This is the principle of stereoscopic vision using image display device 101; by providing a convergence angle B1 with respect to virtual image position 302, the viewer visually perceives an object as if it were located at point 301. However, unlike the real world, in image display device 101, the distance corresponding to convergence angle B1 does not match the distance A2 corresponding to diopter adjustment. This is the so-called contradiction between convergence and diopter adjustment, and is the main cause of viewer nausea and fatigue.

In this embodiment, as shown in FIG. 2, the virtual image position 302 in FIG. 3(B) can be changed by changing the distance between the image display units 201a, 201b and the display optical systems 202a, 202b. For example, assume that the gaze detection means detects that the viewer is gazing at point 301. The distance between the image display unit and the display optical system is changed so that the virtual image position 302, which is at distance A2, becomes distance A1. This causes the distance corresponding to the convergence angle and the distance corresponding to the diopter to match, as shown in FIG. 3(A), allowing for more natural viewing and reducing motion sickness and fatigue. Note that when changing the distance between the image display unit and the display optical system, it is sufficient to drive at least one of the image display unit and the display optical system. In other words, there are embodiments in which the image display unit or the display optical system is driven, and embodiments in which both the image display unit and the display optical system are driven.

Next, the operation of the image display device 101 will be explained using Figures 4 and 5. Figure 4 is a block diagram showing the overall system configuration of the image display device 101 of this embodiment. The arithmetic processing unit 401 of the image display device 101 is equipped with a CPU (central processing unit) and other components, and controls the overall system. Connected to the arithmetic processing unit 401 are image display units 201a and 201b, gaze detection cameras 203a and 203b, infrared illumination units 204a and 204b, and diopter adjustment units 205a and 205b.

Video display device 101 is equipped with a power supply 402, a wireless unit 403, and speakers 404a and 404b. By connecting to a network via wireless unit 403, the user can view video content and the like on video display device 101.

The operation from gaze detection to diopter adjustment will be described with reference to the flowchart in Figure 5. The program corresponding to the flowchart in Figure 5 is stored in the memory unit within the processing unit 401. First, in S501, the gaze of each of the user's right eye 103a and left eye 103b is detected using the gaze detection cameras 203a and 203b and the infrared illumination units 204a and 204b.

Next, in S502, the calculation processing unit 401 executes processing to determine the point of gaze from the line of sight of the right eye 103a and the line of sight of the left eye 103b. The average value of the line of sight of the right eye and the left eye is used to calculate the point of gaze. In S503, the calculation processing unit 401 executes processing to calculate the diopter adjustment amount at the position of the point of gaze of the image displayed on the image display units 201a and 201b. Specifically, the parallax between the image at the position of the point of gaze of the right eye and the image at the position of the point of gaze of the left eye is calculated, and the diopter adjustment amount is derived from this parallax. For example, the image display device 101 has a data table that associates parallax with diopter adjustment amount, and can calculate the diopter adjustment amount corresponding to the parallax. Alternatively, the calculation processing unit 401 can calculate the diopter adjustment amount from the parallax using a mathematical formula that expresses the relationship between parallax and diopter adjustment amount.

In S504, the motors that make up the diopter adjustment units 205a and 205b are driven based on the calculated diopter adjustment amount. As a result, an operation is performed to change the distance between the image display units 201a and 201b and the display optical systems 202a and 202b. Since the above operation is performed every time the gaze point is changed, it is possible to perform diopter adjustment and change the diopter in real time.

The human gaze includes high-frequency micromovements known as fixational eye movements. S501 enables smooth gaze detection by removing the micromovement components. In this embodiment, the diopter adjustment units 205a and 205b can be driven independently for the right and left eyes, making it possible to accommodate users with different visual acuity in each eye. Furthermore, the gaze detection cameras 203a and 203b perform gaze detection without using the display optical systems 202a and 202b. This prevents the display optical system from becoming too large, and enables gaze detection that is less affected by diopter adjustments related to the display optical system to be achieved with a lightweight, compact configuration.

This embodiment makes it possible to provide a smaller, lighter image display device while achieving visibility adjustment according to the position of the gaze point.

[Second Example]
With reference to Figure 6, an image display device according to a second embodiment of the present invention will be described. In this embodiment, a reflective member that serves as an optical path changing means is disposed between the display optical system and the eye in order to reduce the angle formed between the optical axis of the display optical system and the optical axis of the gaze detection camera. This configuration enables gaze detection with higher accuracy. Note that by using the same reference numerals as in the first embodiment, detailed explanations of these elements will be omitted and differences will be mainly explained. This method of omitting explanations will be the same in the embodiments described below.

Figure 6 is a schematic cross-sectional view of the image display device 101 of this embodiment when viewed from the top. The reflecting members 601a and 601b are disposed between the user's eyes 103a and 103b and the display optical systems 202a and 202b, respectively, and serve as optical path changing means.

In Figure 6, the optical axis of the optical system of gaze detection camera 203a relative to the user's right eye 103a is indicated by LGa2. The optical axis of the optical system of gaze detection camera 203b relative to the user's left eye 103b is indicated by LGb2. Gaze detection cameras 203a and 203b are positioned closer to the user than the sides of display optical systems 202a and 202b, respectively, and can capture the user's gaze via reflecting members 601a and 601b. Note that, as with the first embodiment, gaze detection cameras 203a and 203b can capture the user's gaze without going through display optical systems 202a and 202b.

In this embodiment, the angle between optical axis LGa2 and optical axis La is denoted as θ2a, and the angle between optical axis LGb2 and optical axis Lb is denoted as θ2b. Gaze detection camera 203a captures eye 103a from a position tilted at angle θ2a with respect to optical axis La. Similarly, gaze detection camera 203b captures eye 103b from a position tilted at angle θ2b with respect to optical axis Lb. Angles θ2a and θ2b are significantly smaller than the angles θ1a and θ1b shown in the first embodiment. Therefore, perspective distortion occurring in the pupil and corneal reflection images captured by the gaze detection cameras is significantly reduced. This allows for the acquisition of images that more closely resemble the eyeball as viewed from the front, enabling more accurate gaze detection. In particular, when the eyeball rotates away from gaze detection camera 203a or 203b, perspective distortion of the pupil becomes significant. The configuration provided with reflective members 601a and 601b, as in this embodiment, is highly effective in improving gaze detection accuracy.

The gaze detection cameras 203a and 203b perform gaze detection using infrared images generated by the infrared illumination units 204a and 204b. The reflective members 601a and 601b have the property of reflecting infrared light and transmitting visible light. This allows images to be presented to the user while minimizing light loss in the image display units 201a and 201b. Furthermore, in this embodiment, the gaze detection cameras 203a and 203b perform gaze detection without using the display optical systems 202a and 202b, achieving the same effects as in the first embodiment.

[Third Example]
An image display device according to a third embodiment of the present invention will be described with reference to Figure 7. Differences from the second embodiment will be described below. This embodiment uses a liquid crystal optical element in the display optical system that can change the focus by electrical switching. With this configuration, the display optical system itself also serves as a diopter adjustment means, which is effective in reducing the size of the image display device.

Figure 7 is a schematic cross-sectional view of the image display device 101 of this embodiment as seen from the top. Optical elements 701a and 701b constitute the display optical system, and the focal length can be changed by changing the orientation characteristics based on liquid crystal switching. Such optical elements include elements using a Fresnel structure and elements that combine polarization characteristics.

The optical elements 701a and 701b that make up the display optical system not only guide the images on the image display units 201a and 201b to the user's eyes, but also serve as diopter adjustment means. This eliminates the need for a linear drive motor to change the distance between the image display units 201a and 201b and the optical elements 701a and 701b, making it possible to further miniaturize the image display device 101. In this embodiment, the gaze detection cameras 203a and 203b perform gaze detection without using the optical elements 701a and 701b.

In addition to the effects of the second embodiment, this embodiment uses flat optical elements 701a and 701b whose focal length can be changed by an electrical signal, making it possible to realize a more compact image display device 101.

In the above-described embodiment, the optical system does not become larger as in conventional examples, but has a lighter and more compact configuration, and gaze detection can be achieved without being affected by diopter adjustment of the display optical system. In other words, it is possible to achieve smaller size and weight while achieving diopter adjustment according to the position of the gaze point. While preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the spirit thereof.

[Other embodiments]
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

101 Image display device 201a, 201b Image display unit 202a, 202b Display optical system 203a, 203b, 204a, 204b Line-of-sight detection unit 205a, 205b Diopter adjustment unit 601a, 601b Reflecting member (light path changing means)
701a, 701b Optical elements

Claims (7)

first and second image display units for displaying first and second images to the right and left eyes of a user, respectively;
first and second display optical systems corresponding to the first and second image display units, respectively;
first and second line-of-sight detection means for respectively performing line-of-sight detection for the first and second image display units without using the first and second display optical systems;
a calculation means for calculating a diopter adjustment amount corresponding to a parallax between the first image and the second image at a position of a gaze point acquired from detection results of the first and second gaze detection means;
first and second diopter adjustment means for adjusting the diopter of the first and second image display units in accordance with the diopter adjustment amount;
first and second optical path changing means disposed on the user side relative to the first and second display optical systems, respectively;
the first and second line-of-sight detection means each include an infrared illuminator and a camera;
a camera included in the first line-of-sight detection means and a camera included in the second line-of-sight detection means are disposed in a region between an optical axis of the first display optical system and an optical axis of the second display optical system in a direction in which the first and second image display units are aligned ,
the optical axis of the camera included in the first line of sight detection means is inclined at a predetermined angle with respect to the optical axis of the first display optical system,
the optical axis of the camera of the second line of sight detection means is inclined at a predetermined angle with respect to the optical axis of the second display optical system,
the first line-of-sight detection means performs line-of-sight detection via the first optical path changing means,
the second line of sight detection means performs line of sight detection via the second optical path changing means,
the first and second display optical systems are configured with flat optical elements whose focal lengths can be changed by electrical signals;
the first and second display optical systems also function as the first and second diopter adjustment means, respectively;
an optical element constituting the first display optical system is disposed in a region between the first optical path changing means and the first image display unit in a direction of an optical axis of the first display optical system;
an optical element constituting the second display optical system is disposed in a region between the second optical path changing means and the second image display unit in the direction of the optical axis of the second display optical system . 2. The image display device according to claim 1, wherein the first and second image display units respectively display the first and second images having parallax, and perform stereoscopic display via the first and second display optical systems. 3. The image display device according to claim 1 , wherein the first and second optical path changing means have a property of reflecting infrared light and transmitting visible light. 4. The image display device according to claim 1, wherein each of the first and second line-of-sight detection means includes a plurality of the infrared illumination units. 5. The image display device according to claim 1, wherein the camera provided by the first gaze detection means and the camera provided by the second gaze detection means are arranged in an area between the first and second image display units in the direction in which the first and second image display units are aligned. 6. The image display device according to claim 1, which can be worn on the head of a user. a first and second image display unit for displaying a first image and a second image to a right eye and a left eye of a user, respectively; a first and a second display optical system corresponding to the first and second image display unit, respectively; a first and a second line-of-sight detection means for performing line-of-sight detection for the first and second image display unit, respectively; and a first and a second optical path changing means arranged on the user side of the first and second display optical systems, wherein the first and second line-of-sight detection means each include an infrared illumination unit and a camera, and the camera of the first line-of-sight detection means and the camera of the second line-of-sight detection means are arranged in an area between an optical axis of the first display optical system and an optical axis of the second display optical system in a direction in which the first and second image display units are aligned, and the optical axis of the camera of the first line-of-sight detection means is inclined at a predetermined angle with respect to the optical axis of the first display optical system, an optical axis of a camera provided in the line-of-sight detection means is tilted at a predetermined angle with respect to the optical axis of the second display optical system, the first line-of-sight detection means performs line-of-sight detection via the first optical path changing means, the second line-of-sight detection means performs line-of-sight detection via the second optical path changing means, the first and second display optical systems are constituted by flat optical elements whose focal lengths can be changed by an electric signal, the first and second display optical systems also serve as first and second diopter adjustment means, respectively, the optical elements constituting the first display optical system are arranged in a region between the first optical path changing means and the first image display unit in the direction of the optical axis of the first display optical system, and the optical elements constituting the second display optical system are arranged in a region between the second optical path changing means and the second image display unit in the direction of the optical axis of the second display optical system,
a gaze detection step of performing gaze detection for the first and second image display units, respectively, without using the first and second display optical systems;
a calculation step of calculating a diopter adjustment amount corresponding to a parallax between the first image and the second image at a position of a gaze point acquired from a detection result of the line-of-sight detection step;
a diopter adjustment step of adjusting the diopter of the first and second image display units by the first and second diopter adjustment means in accordance with the diopter adjustment amount.