JP7696262B2 - Aerial image display device - Google Patents

Description

This application relates to an aerial image display device.

Aerial image display devices are known that do not require special glasses and can display images in the air without using a screen such as fog. For example, an aerial image display device has been disclosed that uses a beam splitter and a retroreflective sheet to focus an image displayed on an image display unit in space at a position that is plane-symmetrical to the beam splitter, thereby displaying the image (see, for example, Patent Document 1).

JP 2009-25776 A

In one method of using an aerial image display device, a touch panel is displayed as an image displayed in the air (hereafter referred to as the aerial image), and the user operates the touch panel with their finger. Since the aerial image floats in the air, there are no objects around it that can serve as indicators for the position of the user's finger. This creates a problem in that it is difficult for the user to grasp the position of their finger relative to the aerial image.

This application has been made to solve the above-mentioned problems, and aims to provide an aerial image display device that makes it easy for a user to grasp the position of their fingers relative to the aerial image.

The aerial image display device of the present application comprises an image display unit that displays an image , an aerial imaging optical system that images the image displayed on the image display unit in space to display an aerial image, a sensor that detects the position of a user's finger within a spatial region containing the aerial image, and stereo speakers that output sound whose position changes depending on the position of the finger detected by the sensor relative to the aerial image.

The aerial image display device of the present application is equipped with stereo speakers that output sounds whose position changes depending on the position of the finger detected by the sensor relative to the aerial image, making it easier for the user to grasp the position of the finger relative to the aerial image.

1 is a cross-sectional view of an aerial image display device according to a first embodiment. 1 is a schematic diagram of an aerial image display device according to a first embodiment. FIG. 11 is a schematic diagram of an aerial image display device according to embodiment 3.

The following describes in detail an aerial image display device according to an embodiment of the present invention with reference to the drawings.

Embodiment 1.
FIG. 1 is a cross-sectional view of an aerial image display device according to the first embodiment. FIG. 2 is a schematic diagram of the aerial image display device according to the present embodiment. The aerial image display device 1 of the present embodiment has a housing 2, an image display unit 3 arranged inside the housing 2, a beam splitter 4 provided at an opening of the housing 2, and a retroreflective sheet 5 arranged inside the housing 2. Light emitted from the image display unit 3 is incident on the beam splitter 4, and a part of the light is reflected and incident on the retroreflective sheet 5. The retroreflective sheet 5 has a characteristic of reflecting the incident light in the incident direction. The retroreflective sheet 5 reflects the light incident from the beam splitter 4 toward the beam splitter 4. A part of the light reflected from the retroreflective sheet 5 toward the beam splitter 4 passes through the beam splitter 4. The image displayed on the image display unit 3 in this way passes through the beam splitter 4 and is imaged outside the housing 2 to become an aerial image 6. The user 10 can observe this aerial image 6. For example, a touch panel for operating various devices is displayed as the aerial image 6. The user 10 can perform a touch operation on the touch panel by bringing a finger 10a close to the touch panel displayed as the aerial image 6. The position of the finger 10a can be measured by a three-dimensional distance sensor 7 that is provided in the housing 2 and detects the position of an object in a spatial region including the aerial image 6. As the image display unit 3, a liquid crystal display, an organic EL display, a display in which LED elements are arranged, or the like can be used. As the beam splitter 4, a half mirror, a reflective polarizing plate, or the like can be used. As the retroreflective sheet 5, a glass bead type sheet in which spherical transparent glass beads are arranged, a microprism type sheet in which transparent triangular pyramids are arranged, or the like can be used. As the three-dimensional distance sensor 7, for example, a ToF sensor (Time of Flight Sensor), a stereo camera, or the like can be used.
1, the direction perpendicular to aerial image 6 is defined as the z-axis direction, the horizontal direction of aerial image 6 as the x-axis direction, and the direction perpendicular to the z-axis and x-axis directions as the y-axis direction. Hereinafter, the x-axis direction will also be referred to as the left-right direction, the y-axis direction as the up-down direction, and the z-axis direction as the perpendicular direction.

The user 10 can perform a touch operation by bringing a finger 10a close to the touch panel displayed as the aerial image 6. The positional relationship between the position of the finger 10a and the position of the aerial image 6 can be measured by a three-dimensional distance sensor 7 provided on the housing 2. This touch operation is non-contact, so it is possible to prevent contamination of the fingertips due to contact and the spread of harmful substances via the touch panel.

However, because the midair image is suspended in the air, there are no objects around it that can serve as indicators of the position of one's fingers. This creates a problem in that it is difficult for the user to grasp the position of their fingers relative to the midair image.

The aerial image display device 1 of this embodiment makes it easy for the user to grasp the position of their fingers relative to the aerial image. To this end, the aerial image display device 1 of this embodiment includes a control unit 8 that calculates the position of the position of the user's 10 fingers 10a relative to the aerial image 6, detected by the three-dimensional distance sensor 7, and a stereo speaker 9 connected to the control unit 8. The stereo speaker 9 is composed of two speakers arranged along the x-axis direction as shown in FIG. 2. The control unit 8 calculates the position of the user's 10 fingers 10a relative to the aerial image 6, and outputs sounds from the stereo speaker 9 whose localization changes depending on the calculated position. Here, localization refers to the direction of the sound source, i.e., the direction from which the sound is heard, in the sound field of the stereo reproduced sound.

The control unit 8 starts outputting low-frequency bass sounds from the stereo speaker 9 when the distance between the position of the finger 10a of the user 10 and the aerial image 6 in the z-axis direction becomes smaller than a predetermined threshold. At this time, the control unit 8 determines the localization of the sound to be output from the stereo speaker 9 according to the position of the finger 10a of the user 10 relative to the aerial image 6 in the x-axis direction. For example, the control unit 8 localizes the sound to the center when the position of the finger 10a of the user 10 is close to the center of the aerial image 6, and moves the localization of the sound to the right when the position of the finger 10a of the user 10 moves to the right of the aerial image 6. Furthermore, the control unit 8 determines the volume of the sound to be output from the stereo speaker 9 according to the position of the finger 10a of the user 10 relative to the aerial image 6 in the y-axis direction. For example, control unit 8 sets the volume of the sound to a medium volume when the position of finger 10a of user 10 is close to the center of aerial image 6, and increases the volume when the position of finger 10a of user 10 moves to the lower side of aerial image 6. Furthermore, control unit 8 gradually increases the frequency and changes the pitch of the sound output from stereo speakers 9 to a higher pitch as the distance between the position of finger 10a of user 10 and aerial image 6 in the z-axis direction becomes smaller relative to aerial image 6.

In the aerial image display device configured in this manner, the user can recognize the perpendicular distance between the position of their finger and the aerial image from the change in frequency of the sound output from the stereo speakers. The user can also recognize the left-right position of their finger relative to the aerial image from the change in the positioning of the sound output from the stereo speakers. Furthermore, the user can recognize the up-down position of their finger relative to the aerial image from the change in the volume of the sound output from the stereo speakers. Therefore, in the aerial image display device of this embodiment, it is easy for the user to grasp the position of their finger relative to the aerial image. Also, because the sound changes in a variety of ways depending on the movement of their finger, the user can get the feeling that they are operating the aerial image display device, improving the sense of operation.

The control unit 8 may discontinuously change the frequency of the sound output from the stereo speaker 9 when the position of the finger coincides with the position of the air image. As another method, the control unit 8 may stop the sound output from the stereo speaker 9 when the position of the finger coincides with the position of the air image. By controlling in this way, the user can recognize that his/her finger has reached the position of the air image. The coincidence of the finger position with the position of the air image is not limited to the time when the distance between the finger position and the position of the air image becomes completely zero, but also includes the time when the distance is smaller than a predetermined distance. The control unit 8 may also generate a special sound as a touch sound when the user's finger touches a button on the touch panel displayed in the air image. The frequency of the sound output from the stereo speaker 9 decreases when the user's finger passes through the air image, so the user can also recognize that his/her finger has passed the position of the air image. The sound output from the stereo speaker 9 may be a discordant sound when the user's finger passes through the air image.

In this embodiment, the control unit 8 gradually increases the frequency as the perpendicular separation distance decreases, changing the sound output from the stereo speakers to a higher pitch, but it may also gradually decrease the frequency as the separation distance decreases, changing the sound output from the stereo speakers to a lower pitch. Also, the control unit 8 increases the volume as the position of the user's 10 finger 10a moves downward relative to the aerial image 6, but it may also decrease the volume as the position of the user's 10 finger 10a moves downward relative to the aerial image 6.

Embodiment 2.
In the aerial image display device of the first embodiment, the control unit causes the stereo speakers to output sounds whose frequencies change continuously according to the distance between the position of the user's finger and the aerial image in the perpendicular direction. In the aerial image display device according to the second embodiment, the control unit causes the stereo speakers to output sounds whose intermittent sounds change continuously according to the distance between the position of the user's finger and the aerial image in the perpendicular direction. The configuration of the aerial image display device of this embodiment is similar to the configuration of the aerial image display device of the first embodiment shown in Figs. 1 and 2. Similarly to the first embodiment, the control unit changes the localization of the sounds output from the stereo speakers according to the position of the user's finger relative to the aerial image in the horizontal direction, and changes the volume of the sounds output from the stereo speakers according to the position of the user's finger relative to the aerial image in the vertical direction.

For example, when the finger 10a of the user 10 approaches the aerial image 6 and the distance in the z-axis direction becomes smaller than a predetermined threshold, the control unit 8 starts outputting intermittent sounds from the stereo speaker 9. At this time, the sound output from the stereo speaker 9 is a sound of a certain length that is generated with a stop period between, and the control unit 8 gradually shortens the stop period as the distance becomes smaller. Also, as in the first embodiment, the control unit 8 changes the localization of the sound output from the stereo speaker 9 according to the position of the finger 10a of the user 10 in the x-axis direction relative to the aerial image 6 and the position of the aerial image 6. Furthermore, the control unit 8 changes the volume of the sound output from the stereo speaker 9 according to the position of the finger 10a of the user 10 in the y-axis direction relative to the aerial image 6 and the position of the aerial image 6.

In the aerial image display device configured in this manner, the user can recognize the perpendicular distance between the position of their finger and the aerial image from the change in the pause period of the sound output from the speaker. The user can also recognize the left-right position of their finger relative to the aerial image from the change in the positioning of the sound output from the stereo speakers. Furthermore, the user can recognize the up-down position of their finger relative to the aerial image from the change in the volume of the sound output from the stereo speakers. Therefore, in the aerial image display device of this embodiment, it is easy for the user to grasp the position of their finger relative to the aerial image. Also, because the sound changes in a variety of ways depending on the movement of their finger, the user can get the feeling that they are operating the aerial image display device, improving the sense of operation.

In this embodiment, the sound output from the stereo speakers 9 is a sound that is generated with a constant length of sound separated by a stop period, and the stop period gradually becomes shorter as the perpendicular distance becomes smaller. As an alternative, the stop period may be constant, and the length of the generation period during which the sound is generated may be gradually shortened as the separation distance becomes smaller. As a further alternative, the cycle at which the intermittent sound generation starts may be constant, and the period during which the sound is generated within that cycle may be gradually changed as the separation distance becomes smaller. In other words, the aerial image display device of this embodiment is configured so that at least one of the stop period and generation period of the intermittent sound changes continuously according to the perpendicular distance between the position of the user's finger and the position of the aerial image.

The control unit 8 may discontinuously change at least one of the stop period and the generation period of the sound output from the stereo speaker 9 when the position of the finger and the position of the air image match. As another method, the control unit 8 may change the frequency of the sound output from the stereo speaker 9 when the position of the finger and the position of the air image match. As a further method, the control unit 8 may stop the sound output from the stereo speaker 9 when the position of the finger and the position of the air image match. By controlling in this manner, the user can recognize that his/her finger has reached the position of the air image. Note that the match between the position of the finger and the position of the air image is not limited to when the distance between the position of the finger and the position of the air image becomes completely zero, but also includes the case where the distance is smaller than a predetermined distance. In addition, the control unit 8 may sound a special sound as a touch sound when the user's finger touches a button on the touch panel displayed in the air image. Note that when the user's finger passes through the air image, the stop period or generation period of the sound output from the stereo speaker 9 becomes longer, so the user can also recognize that his/her finger has passed the position of the air image. Additionally, when the user's finger passes through the air image, the sound output from the stereo speaker 9 may be a discordant sound.

Embodiment 3.
In the aerial image display devices of the first and second embodiments, the control unit changes the localization of the sound output from the stereo speakers according to the position of the user's finger and the position of the aerial image in the x-axis direction, and changes the volume of the sound output from the stereo speakers according to the position of the user's finger and the position of the aerial image in the y-axis direction. In the aerial image display device of the third embodiment, a stereo speaker array is used to change the localization of the sound in the xy plane output from the stereo speaker array according to the position of the user's finger on the xy plane.

Figure 3 is a schematic diagram of an aerial image display device according to this embodiment. The aerial image display device 1 of this embodiment is the aerial image display device of embodiment 1, which is equipped with a pair of stereo speaker arrays 11 as stereo speakers. The stereo speaker array 11 is composed of two speaker arrays arranged along the x-axis direction as shown in Figure 3. The multiple speakers that make up each speaker array are arranged along a direction parallel to the y-axis direction. Therefore, the stereo speaker array 11 can change the positioning of sound in the xy plane.

When the distance between the position of the finger 10a of the user 10 and the aerial image 6 in the z-axis direction relative to the aerial image 6 becomes smaller than a predetermined threshold, the control unit 8 starts outputting low-frequency bass sounds from the stereo speaker array 11. At that time, the control unit 8 changes the localization in the xy plane of the sound output from the stereo speaker array 11 according to the position of the finger 10a of the user 10 and the position of the aerial image 6 in the xy plane relative to the aerial image 6. When the position of the finger 10a of the user 10 is close to the center of the aerial image 6, the control unit 8 localizes the sound to the center, and when the position of the finger 10a of the user 10 moves in the xy plane of the aerial image 6, the control unit 8 moves the localization of the sound according to its position. In addition, the control unit 8 gradually increases the frequency as the distance between the position of the finger 10a of the user 10 and the aerial image 6 in the z-axis direction relative to the aerial image 6 becomes smaller, changing the sound output from the stereo speaker 9 to a higher pitch.

Note that the control unit 8 changes the frequency depending on the distance between the position of the finger 10a of the user 10 in the z-axis direction relative to the aerial image 6 and the aerial image 6, but instead of changing the frequency, the period of the intermittent sound may be changed. The control unit 8 of the aerial image display device of this embodiment changes the sound quality of the sound output from the stereo speaker array 11 depending on the distance between the position of the finger 10a of the user 10 in the z-axis direction relative to the aerial image 6 and the aerial image 6.

In an aerial image display device configured in this manner, the user can recognize the distance in the z-axis direction between the position of the aerial image and the position of their finger from the change in sound quality of the sound output from the speaker. Also, the user can recognize the position of their finger in the xy plane relative to the aerial image from the change in the positioning in the xy plane of the sound output from the stereo speaker array. Therefore, in the aerial image display device of this embodiment, it is easy for the user to grasp the position of their finger in relation to the aerial image. Also, because the sound changes in a variety of ways depending on the movement of the user's finger, the user gets the feeling that they are operating the aerial image display device, improving the sense of operation.

In this embodiment, a stereo speaker array is used to change the position of the sound in the xy plane, but three or more speakers may be used instead of stereo speakers to change the position of the sound in the xy plane.

In embodiments 1 to 3, the position of finger 10a is measured by three-dimensional distance sensor 7. The aerial image display device of embodiments 1 to 3 may include a recognition unit for calculating the position of the fingertip that will touch the touch panel from the position of finger 10a measured by three-dimensional distance sensor 7. Furthermore, in embodiments 1 to 3, it is also assumed that the displayed image will be switched in response to a touch operation when the aerial image is used as a touch panel. The aerial image display device of embodiments 1 to 3 may include a display processing unit for switching the displayed image on the image display unit based on the fingertip position calculated by the recognition unit.

In addition, in the first to third embodiments, the aerial image display device has been described using a system that uses a beam splitter and a retroreflective sheet. Other aerial image display devices include a system that uses an optical plate called a dihedral corner reflector array and a system that uses an optical plate called an orthogonal mirror array. In these aerial image display devices, the retroreflective sheet is removed from the aerial image display device shown in FIG. 1, and an optical plate is placed at the position of the beam splitter. This type of aerial image display device can display an aerial image by focusing an image displayed on the image display unit in a space that is plane-symmetrical to the optical plate. Even with this type of aerial image display device, there is a problem that it is difficult to grasp the position of the user's fingers relative to the aerial image. Even with this type of aerial image display device, by providing stereo speakers that output sounds whose position changes depending on the position of the user's fingers relative to the aerial image, it becomes easier to grasp the position of the user's fingers relative to the aerial image.

Although the present application describes various exemplary embodiments and examples, the various features, aspects, and functions described in one or more embodiments are not limited to application to a particular embodiment, but may be applied to the embodiments alone or in various combinations.
Therefore, countless modifications not exemplified are assumed within the scope of the technology disclosed in the present specification, including, for example, modifying, adding, or omitting at least one component, and further, extracting at least one component and combining it with a component of another embodiment.

1 Aerial image display device, 2 Housing, 3 Image display unit, 4 Beam splitter, 5 Retroreflective sheet, 6 Aerial image, 7 Three-dimensional distance sensor, 8 Control unit, 9 Stereo speaker, 10 User, 10a Finger, 11 Stereo speaker array.

Claims (12)

a video display unit for displaying a video ;
an aerial imaging optical system that images the image displayed on the image display unit in space to display an aerial image;
a sensor for detecting a position of a user's finger within a spatial region including the aerial image;
and stereo speakers that output sounds whose localization changes depending on the position of the finger detected by the sensor relative to the aerial image. The aerial image display device described in claim 1, characterized in that the aerial imaging optical system is composed of a beam splitter that reflects the image displayed on the image display unit, and a retroreflective sheet that retroreflects the image reflected by the beam splitter. The aerial imaging optical system is composed of an optical plate that images the image displayed on the image display unit in space to display the aerial image, and the optical plate images the image displayed on the image display unit in space at a position plane-symmetrical to the optical plate to display the aerial image. The aerial image display device described in claim 1, characterized in that the aerial imaging optical system is composed of an optical plate that images the image displayed on the image display unit in space at a position plane-symmetrical to the optical plate to display the aerial image. 4. The aerial image display device according to any one of claims 1 to 3, further comprising a control unit that controls sound to be output from the stereo speakers in accordance with a position of the finger detected by the sensor relative to the aerial image, wherein the control unit changes sound quality of the sound output from the stereo speakers in accordance with a distance between the position of the finger and the aerial image in a direction perpendicular to the aerial image, and changes a position of the sound output from the stereo speakers in accordance with a position of the finger relative to the aerial image in a direction parallel to the aerial image. The aerial image display device according to claim 4 , wherein the frequency of the sound output from the stereo speakers changes continuously depending on the distance between the position of the finger in a direction perpendicular to the aerial image and the aerial image. The aerial image display device according to claim 5 , wherein the frequency of the sound output from the stereo speakers changes discontinuously when the position of the finger and the position of the aerial image coincide with each other. The aerial image display device according to claim 4, characterized in that the sounds output from the stereo speakers are intermittent sounds, and at least one of a stop period and a generation period of the intermittent sounds changes continuously depending on a distance between the position of the finger in a direction perpendicular to the aerial image and the aerial image. The aerial image display device according to claim 7, characterized in that at least one of the stop period and the generation period of the sound output from the stereo speakers changes discontinuously when the position of the finger and the position of the aerial image coincide with each other. The floating image display device according to claim 5 or 7 , wherein sounds output from the stereo speakers stop when the position of the finger and the position of the floating image coincide with each other. The floating image display device according to any one of claims 4 to 9 , wherein sounds output from the stereo speakers become discordant when the position of the finger passes through the floating image. The aerial image display device described in any one of claims 4 to 10, characterized in that the control unit changes a position of the sound output from the stereo speakers in accordance with a position of the finger relative to the aerial image in one direction parallel to the aerial image, and changes a volume of the sound output from the stereo speakers in accordance with a position of the finger relative to the aerial image in a direction perpendicular to the one direction. 12. The aerial image display device according to claim 1 , wherein the stereo speakers are configured by a pair of speaker arrays.

Images