JP4619216B2 - Stereoscopic image display apparatus and stereoscopic image display method - Google Patents

Description

  The present invention relates to a stereoscopic image display apparatus and a stereoscopic image display method for displaying a display object as a stereoscopic image.

  Conventionally, in a mobile communication terminal, a 3D display using a stereo image display method (monocular stereo viewing method) is mainly used as a display for displaying a display object as a stereoscopic image.

  Such a stereo image display method is a method for projecting different images (parallax images) to the left and right eyes of a human, and is specifically realized using a parallax barrier or a lenticular sheet.

  In such a stereo image display method, as shown in FIG. 6, an image projected to the left eye (left eye image) and an image projected to the right eye (right eye image) are displayed on a normal 2D display. Therefore, the stereo image display method has a problem that the position where the observer can observe the stereoscopic image is limited.

  In addition, since the resolution (number of pixels) of the 2D display is physically limited, in the case of a stereo image display method, the resolution of the left and right parallax images is 1/2 the number of pixels, and multi-viewpoint image display having N viewpoints is performed. In the case of the system (multi-view display), the parallax image at each viewpoint has a 1 / N number of pixels, and the resolution of the stereoscopic image is deteriorated.

  In recent years, the display resolution has been improved, and a display capable of observing a stereoscopic image from a plurality of viewpoints has been developed. However, in the portable stereoscopic image display device (mobile communication terminal), the physical limit of the screen size still remains.

  On the other hand, in a desktop stereoscopic image display device, by limiting to only the parallax in the horizontal direction, images of a plurality of viewpoints in the horizontal direction are displayed with high density, and when the observer moves the head in the horizontal direction, the visible image is displayed. A method (high-density directional image display method) capable of realizing motion parallax such as switching naturally has been proposed (see Non-Patent Document 1, for example).

  Such a high-density directional image display method is different from the conventional stereo image display method in that the position at which a stereoscopic image can be observed is not limited, and natural motion parallax can be realized although only in the horizontal direction.

  In such a high-density directional image display method, for example, as shown in FIG. 7, images of a plurality of viewpoints (for example, four viewpoints “a”, “b”, “c”, “d”) are displayed in the horizontal and vertical directions. The image dispersed in the screen is displayed on a conventional 2D display.

  The lenticular sheet 20 attached obliquely has a function of projecting images displayed with their vertical positions shifted in parallel in the same horizontal direction. Specifically, as illustrated in FIG. 7, the lenticular sheet 20 includes pixels corresponding to the images of the viewpoints “a” and “b” and pixels corresponding to the images of the viewpoints “c” and “d”. It is configured to perform parallel projection in the same horizontal direction.

  That is, in the high-density directional image display method, the pixels corresponding to the images of the viewpoints “a”, “b”, “c”, and “d” are projected in the same horizontal direction and in different gaze directions. Thus, the observer can sequentially observe images of different viewpoints in the horizontal direction.

As described above, according to the high-density directional image display method, it is possible to display a stereoscopic image with smooth motion parallax by arranging images of different viewpoints at high density.
Ebisawa and Takagi, “Thin 3D Display Displaying 72 Directional Images”, 3D Image Conference 2004, pp. 17-20, 2004

  However, the above-described high-density directional image display method can cope with the parallax in the horizontal direction (viewpoint change), but cannot cope with the parallax in the vertical direction as shown in FIG.

  Therefore, there is a problem that the observer can always observe only the image of the same viewpoint even when the head is moved up and down or the stereoscopic image display device is tilted back and forth.

  Therefore, the present invention has been made in view of the above points, and provides a stereoscopic image display apparatus and a stereoscopic image display method that can realize horizontal and vertical motion parallax and display a natural stereoscopic image. The purpose is to do.

  A first feature of the present invention is a stereoscopic image display device that displays a display object as a stereoscopic image, and a three-dimensional information storage unit configured to store three-dimensional information of the display object; Multi-viewpoint image generation configured to generate a multi-viewpoint image based on a plurality of images obtained by observing the display target object from a plurality of viewpoints in a first direction with reference to the three-dimensional information of the display target object A line-of-sight direction determining unit configured to determine a line-of-sight direction in a second direction of an observer observing the stereoscopic image, and the multi-viewpoint image selected based on the line-of-sight direction of the observer And a stereoscopic image display unit configured to display the image.

  In the first feature of the present invention, the line-of-sight direction determination unit may be configured to determine the line-of-sight direction of the observer by detecting an inclination of the stereoscopic image display unit.

  1st characteristic of this invention WHEREIN: The said gaze direction determination part may be comprised so that the inclination of the said stereo image display part may be detected using an acceleration sensor.

  In the first feature of the present invention, the line-of-sight direction determining unit is configured to determine the line-of-sight direction of the observer based on an image captured by a camera attached to the stereoscopic image display device. Also good.

  In the first feature of the present invention, the line-of-sight direction determination unit detects the position of the observer by the camera, and from the relationship between the detected position of the observer and the position of the stereoscopic image display unit, You may be comprised so that an observer's gaze direction may be calculated.

  1st characteristic of this invention WHEREIN: The said gaze direction determination part detects the movement rotation information of the said stereoscopic image display apparatus from the image imaged by the said camera, Based on the said movement rotation information, the said gaze direction of the said observer May be configured to update.

  A second feature of the present invention is a stereoscopic image display method for displaying a display object as a stereoscopic image, the stereoscopic image display method for displaying the display object as a stereoscopic image, wherein the stored display object A step of generating a multi-viewpoint image based on a plurality of images obtained by observing the display object from a plurality of viewpoints in a first direction with reference to the three-dimensional information of the object; The gist of the present invention is to include a step of determining a line-of-sight direction in two directions and a step of displaying the multi-viewpoint image selected based on the line-of-sight direction of the observer.

  As described above, according to the present invention, it is possible to provide a stereoscopic image display apparatus and a stereoscopic image display method that can realize motion parallax in the horizontal direction and the vertical direction and display a natural stereoscopic image.

(Configuration of stereoscopic image display device according to first embodiment of the present invention)
The configuration of the stereoscopic image display apparatus 1 according to the first embodiment of the present invention will be described with reference to FIGS. The stereoscopic image display device 1 according to the present embodiment is configured to display a display object as a stereoscopic image. In this specification, an image includes both a still image and a moving image.

  As shown in FIG. 1, the stereoscopic image display apparatus 1 according to the present embodiment includes a 3D information accumulation unit 10, a multi-viewpoint image generation unit 11, a stereoscopic image display unit 12, and a line-of-sight direction detection unit 13. ing.

  The 3D information storage unit 10 is configured to store three-dimensional information (hereinafter, 3D information) of a display object. For example, the 3D information storage unit 10 stores and holds 3D information (for example, multi-camera input data, 3D polygon data, etc.) acquired by some means, and outputs an arbitrary viewpoint image to the stereoscopic image display unit 12. It is configured as follows.

  The multi-viewpoint image generation unit 11 refers to the 3D information of the display object held in the 3D information storage unit 10 and observes the display object from a plurality of viewpoints in the horizontal direction (first direction). The multi-viewpoint image is generated based on

  That is, the multi-viewpoint image generation unit 11 acquires images from a plurality of viewpoints from the 3D information held in the 3D information storage unit 10 and converts them into a single image to display on the stereoscopic image display unit 12. A multi-viewpoint image having an image format for generating a multi-viewpoint image.

  Specifically, the multi-viewpoint image generation unit 11 has a predetermined angle interval (for example, an interval of 1 degree) in the horizontal direction around the viewpoint specified in advance or the viewpoint given by an instruction from the observer. Images from different viewpoints are acquired from the 3D information storage unit 10, and the pixels of the acquired images are rearranged into a pixel array corresponding to the stereoscopic image display unit 12, thereby generating a multi-viewpoint image.

  Further, the multi-viewpoint image generation unit 11 calculates a gaze direction corresponding to the tilt angle notified from the gaze direction detection unit 13, and determines in advance based on the plurality of images and gaze directions read from the 3D information storage unit 10. A multi-viewpoint image reconstructed with the specified image format is generated.

  Specifically, the multi-view image generation unit 11 is configured to rotate the generated multi-view image based on the calculated line-of-sight direction.

  For example, when the stereoscopic image display unit 12 is tilted as illustrated in FIG. 3B (that is, when the line-of-sight direction is changed), the multi-viewpoint image generation unit 11 observes the display object from the line-of-sight direction-1. The multi-viewpoint image is rotated so as to become a viewpoint image.

  Similarly, when the stereoscopic image display unit 12 is tilted as illustrated in FIG. 3C (that is, when the line-of-sight direction is changed), the multi-viewpoint image generation unit 11 observes the display object from the line-of-sight direction + 1. The multi-viewpoint image is rotated so as to become a viewpoint image.

  In this regard, in the conventional high-density directional image display method, as shown in FIGS. 3D to 3F, the observer always observes only images from the same line-of-sight direction even when the line-of-sight direction changes. I can't.

  The multi-viewpoint image generation unit 11 is configured to sequentially generate a multi-viewpoint image every time the tilt of the stereoscopic image display device 1 is detected.

  In addition, the multi-viewpoint image generation unit 11 is configured to output the generated multi-viewpoint image to the stereoscopic image display unit 12.

  The line-of-sight direction detection unit 13 is configured to determine the line-of-sight direction in the vertical direction (second direction) of the observer who observes the stereoscopic image.

  For example, as illustrated in FIG. 3, when the gaze direction detection unit 13 detects the tilt of the stereoscopic image display unit 12, the gaze direction detection unit 13 detects a tilt angle with respect to the gaze direction 0 and notifies the multi-viewpoint image generation unit 12. It is configured as follows.

  In the case of FIG. 3B, the line-of-sight direction detection unit 13 detects an angle between the line-of-sight direction 0 and the line-of-sight direction −1 as the tilt angle, and in the case of FIG. 3C, the line-of-sight direction 0 and the line-of-sight direction. An angle between +1 is detected as the tilt angle.

  Further, the line-of-sight direction detection unit 13 is configured to determine the line-of-sight direction of the observer by detecting the inclination of the stereoscopic image display unit 12. Here, the line-of-sight direction determination unit 13 may be configured to detect the inclination of the stereoscopic image display unit 12 by using an acceleration sensor.

  Note that the line-of-sight direction determination unit 13 may be configured by a detection unit of an acceleration sensor as long as it is a unit that can determine the inclination of the stereoscopic image display device 1 or the line-of-sight direction of the observer.

  For example, in a notebook PC display, it is conceivable to provide a physical angle detection means for detecting an angle between the display and a keyboard or the like as the line-of-sight direction determination unit 12.

  The line-of-sight direction determination unit 13 may be configured to determine the line-of-sight direction of the observer based on an image captured by a camera attached to the stereoscopic image display device 1.

  Specifically, the line-of-sight direction determination unit 13 detects the position of the observer using the camera, and calculates the line-of-sight direction of the observer from the relationship between the detected position of the observer and the position of the stereoscopic image display unit 12. It may be configured to.

  The line-of-sight direction determination unit 13 is configured to detect movement / rotation information of the stereoscopic image display device 1 from an image captured by the camera and update the line-of-sight direction of the observer based on the movement / rotation information. May be.

  For example, the gaze direction determination unit 13 detects the position of the observer's head, the position of the observer's face, the position of the observer's eyes, or the like from the image captured by the camera, and is obtained by image processing. It may be configured to calculate the viewing direction of the observer from the feature amount.

  For example, when there is one camera attached to the stereoscopic image display device 1, the line-of-sight direction determination unit 13 determines the center position of the observer's face or the (x, y) coordinates of the eye position by the image processing. Estimate from the obtained feature value.

The line-of-sight direction determination unit 13 presupposes a distance (z coordinate, for example, 40 cm) between the eyes of the observer and the stereoscopic image display unit (display) 12 in advance, and displays the center coordinates ((x d , Y d , z d ) coordinates) and the position of the observer's eyes ((x, y, zd + 40 ) coordinates) may be used to estimate the viewer's line-of-sight direction.

  When there are two cameras attached to the stereoscopic image display device 1, the line-of-sight direction determination unit 13 may estimate the three-dimensional position or orientation of the observer's head by stereo measurement.

  In addition, the line-of-sight direction determination unit 13 estimates the camera movement (moving and rotating operation) with respect to the image captured by the camera using an optical flow or the like, and calculates the orientation of the display itself from the camera movement. The image observed from the line-of-sight direction that matches the direction of the display may be displayed.

  Further, a 3D position sensor may be attached to the stereoscopic image display device 1 and an image displayed on the stereoscopic image display unit 12 may be updated in accordance with the movement of the camera.

  Specifically, the line-of-sight direction determination unit 13 extracts feature points from the image captured by the camera, calculates the movement of the stereoscopic image display device 1 from the trajectory of the feature points, and uses the trajectory of the feature points. It may be configured to synchronize and update the stereoscopic image displayed on the stereoscopic image display unit 12.

  Further, the line-of-sight direction determination unit 13 may be configured to determine the line-of-sight direction of the observer according to an instruction from the observer.

  The stereoscopic image display unit 12 is configured to display a multi-viewpoint image determined based on the viewer's line-of-sight direction determined by the line-of-sight direction determination unit 13. Specifically, the stereoscopic image display unit 12 is configured to display the multi-viewpoint image transmitted from the multi-viewpoint image generation unit 11.

  Even when the observer moves his / her head in the horizontal direction, the stereoscopic image display unit 12 can observe images of different viewpoints in the horizontal direction and can perceive motion parallax. Note that the stereoscopic image display unit 12 may be provided with an optical system capable of observing a multi-viewpoint image from each viewpoint.

  Further, the stereoscopic image display unit 12 may be configured to display an image of each viewpoint in the horizontal direction by the high-density directional image display method described in Non-Patent Document 1 described above.

  In such a case, as shown in FIG. 4, the stereoscopic image display unit 12 realizes motion parallax by allowing a viewer to observe a directional image in the horizontal direction, and in the vertical direction, the stereoscopic image display device 1. Motion parallax is realized by changing the image to be displayed by tilting.

  FIG. 2 shows an example of the stereoscopic image display apparatus 1 according to the present embodiment. As illustrated in FIG. 2, the 3D information storage unit 10 and the multi-viewpoint image generation unit 11 are mounted on the processor 14. The line-of-sight direction detection unit 13 is constituted by a small acceleration sensor. In addition, the stereoscopic image display device 1 according to the present embodiment may be configured to include speakers 15 and 16 that can reproduce a 3D sound field.

  In addition, the stereoscopic image display unit 12 is configured by a translucent display, and it is conceivable to display a stereoscopic image corresponding to a display object that can be seen through the display. That is, a camera may be attached to the back surface of the stereoscopic image display device 1 so that the same image as the image observed through the display can be acquired by the camera.

(Operation of the stereoscopic image display apparatus according to the first embodiment of the present invention)
With reference to FIG. 5, the operation of the stereoscopic image display apparatus 1 according to the present embodiment will be described.

  As shown in FIG. 5, when the observer tilts the stereoscopic image display device 1 in step S <b> 101, the line-of-sight direction detection unit 13 detects the inclination of the stereoscopic image display unit 12 of the stereoscopic image display device 1.

  In step S <b> 102, the line-of-sight direction detection unit 13 determines the line-of-sight direction of the observer who observes the display target based on the inclination of the stereoscopic image display unit 12.

  In step S103, the multi-viewpoint image generation unit 11 acquires 3D information of the display object from the 3D image storage unit 10, and in step S104, based on the acquired 3D information and the observer's line-of-sight direction, displays a stereoscopic image. A multi-viewpoint image to be displayed on the unit 12 is generated.

  The stereoscopic image display unit 12 displays the multi-viewpoint image generated by the multi-viewpoint image generation unit 11. Here, the multi-viewpoint image is generated based on a plurality of images obtained by observing the display object from a plurality of viewpoints in the horizontal direction (see FIG. 7). Further, the multi-viewpoint image is different for each determined line-of-sight direction of the observer.

(Operation and effect of the stereoscopic image display apparatus according to the first embodiment of the present invention)
Generally, since a person moves his / her head left and right unconsciously, if there is a delay in the change in the image of each viewpoint in the horizontal direction, a sense of incongruity appears. On the other hand, in the vertical direction, people often consciously move their heads or consciously move the display object consciously in their hands, which delays the change in the image of each viewpoint in the vertical direction. Even if this occurs, it is possible to reduce the uncomfortable feeling by feedback such as slowing the movement of the head or slowing the movement of the hand.

  Therefore, according to the stereoscopic image display apparatus 1 according to the present embodiment, smooth motion parallax is realized by using a conventional high-density directional image display method in the horizontal direction, and the line-of-sight direction of the observer in the vertical direction. Since the stereoscopic image (multi-viewpoint image) to be displayed is changed based on the image, the observer can freely view images of different viewpoints in the upper, lower, left, and right directions. It is possible to observe the display object as if looking into it from a free viewpoint.

1 is a functional block diagram of a stereoscopic image display device according to a first embodiment of the present invention. It is a figure which shows an example of the three-dimensional image display apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the gaze direction detection part and multiview image display part of the stereo image display apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the stereo image display part by the stereo image display apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the stereo image display apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the conventional stereo image display system. It is a figure for demonstrating the conventional multiview image display system. It is a figure for demonstrating the problem of the conventional multiview image display system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Stereoscopic image display apparatus 10 ... 3D information storage part 11 ... Multi-viewpoint image generation part 12 ... Stereoscopic image display part 13 ... Gaze direction detection part 14 ... Processor 15, 16 ... Speaker

Claims (2)

A stereoscopic image display device that displays a display object as a stereoscopic image,
A 3D information storage unit configured to store 3D information of the display object;
Multi-viewpoint image generation configured to generate a multi-viewpoint image based on a plurality of images obtained by observing the display target object from a plurality of viewpoints in a first direction with reference to the three-dimensional information of the display target object And
A line-of-sight direction determining unit configured to determine a line-of-sight direction in a second direction of an observer observing the stereoscopic image;
A stereoscopic image display unit configured to display the multi-viewpoint image selected based on the viewing direction of the observer ;
The line-of-sight direction determination unit presupposes a distance between the observer's eyes and the stereoscopic image display unit, and from the center coordinates of the stereoscopic image display unit and the position of the observer's eyes, A stereoscopic image display device characterized by determining a viewing direction of an observer . A stereoscopic image display method for displaying a display object as a stereoscopic image,
Generating a multi-viewpoint image based on a plurality of images obtained by observing the display target object from a plurality of viewpoints in a first direction with reference to the stored three-dimensional information of the display target object;
Determining a line-of-sight direction in a second direction of an observer observing the stereoscopic image;
Displaying the multi-viewpoint image selected based on the viewing direction of the observer,
In the step of determining the line-of-sight direction, a distance between the observer's eyes and the stereoscopic image display unit is assumed in advance, and the center coordinates of the stereoscopic image display unit and the position of the observer's eyes are determined. A stereoscopic image display method , wherein the viewing direction of the observer is determined .

Images